Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l5kVj-0001IE-4N for pgsql-hackers@arkaria.postgresql.org; Sat, 30 Jan 2021 07:17:36 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1l5kVB-00036d-2l for pgsql-hackers@arkaria.postgresql.org; Sat, 30 Jan 2021 07:17:01 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l5kV9-00036S-Qz for pgsql-hackers@lists.postgresql.org; Sat, 30 Jan 2021 07:17:00 +0000 Received: from mail-io1-xd34.google.com ([2607:f8b0:4864:20::d34]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1l5kUu-0005AN-2d for pgsql-hackers@lists.postgresql.org; Sat, 30 Jan 2021 07:16:58 +0000 Received: by mail-io1-xd34.google.com with SMTP id s24so259003iob.6 for ; Fri, 29 Jan 2021 23:16:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=dbnOV+JeMIb7iMuyUK7oBoNsrq6QKUYfhXkTg6MqRzY=; b=1r4Mu9f0rufR7/9mmtbXY5u6sf6i2zbLHUKXASdwL/NeYiva/d3NgTW6InUtkXRmK5 dQMarfflKr24IsEHuCqPPq0SPdu4Xgo3zmNhjnJUL/fiY7o3Gma9Rf5B/9dzbXvpHaem cJ0/u5d+quJ5+U6oJ01lbePXtWy4g1iIYb+ddwlT6hUXy2prmQ8QSVmilrrBtCg2HRhW W832sLsLZTuMqsna6wcVy6AG+WMKrAG+cbH8ScNiICTkF2LZYOySeNK8nRxb3beW8V0x Rne1xAa2XcggnmsXewhXcvE7yLB3zukhApfR64dpkA2Qwa8Q38APIX21butvQEVphVPw iAcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=dbnOV+JeMIb7iMuyUK7oBoNsrq6QKUYfhXkTg6MqRzY=; b=SH4aA28DX4FbAGCL1izlotfuao7rsIpKlEaNPEX7vy4xsd331/KrmcBq9m6mj7e02R snqyqibaiO5sT+H+snNv8s/aAdeFnd+28qLCNt6/EP4GTPCJZtyoJH9N5Y/znfe0ZS1P yCWi2OG5ZqDnAtJpBr9HS3iiOoZ/PTSyCbo/UsQCOzdL1+K0d2NmTWJ930DE0bfRtX9Q ELes8+9r2hwvRlvKKbO52yN1F/xwcifttm+6zdZZemryVgoCZfRt/exo51d8AEs6Y+wz NORNfFfEBhMEh9EZHBb4ZDyZuittjPaAIlFaGsBgYiTJ3ZrSyRL27EJ6Fk1WtdTG0gKx 1xgQ== X-Gm-Message-State: AOAM531wwYzFwzgH0819bvE9J3CkmiTUg+r4cxVdhlrAWbu5t97vrjk8 vElFuxHO+Qlk4JbV2SxQWQ7/7Q== X-Google-Smtp-Source: ABdhPJzFLtPiQ4SCF5Y1I3AnAr1rQlsbGpLCtZ0xsliNth7O6/6XiLopkzPPE29XrDudPNHoSeclGA== X-Received: by 2002:a6b:d80e:: with SMTP id y14mr7261283iob.101.1611991002846; Fri, 29 Jan 2021 23:16:42 -0800 (PST) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id x7sm5300528ioa.48.2021.01.29.23.16.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jan 2021 23:16:40 -0800 (PST) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id 27DCC8009D6; Sat, 30 Jan 2021 01:16:39 -0600 (CST) Date: Sat, 30 Jan 2021 01:16:39 -0600 From: Justin Pryzby To: Dilip Kumar Cc: Robert Haas , Tomas Vondra , Alexander Korotkov , David Steele , Ildus Kurbangaliev , Dmitry Dolgov <9erthalion6@gmail.com>, pgsql-hackers@lists.postgresql.org Subject: Re: [HACKERS] Custom compression methods Message-ID: <20210130071638.GG7450@telsasoft.com> References: <20210119190720.GL8560@telsasoft.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="ytoMbUMiTKPMT3hY" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --ytoMbUMiTKPMT3hY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline > + * Helper function for ATExecSetStorage and ATExecSetCompression > + * > + * Set the attcompression or attstorage for the respective index attribute. If > + * attcompression is a valid oid then it will update the attcompression for the > + * index attribute otherwise it will update the attstorage. > + */ > +static void > +ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum, > + Oid newcompression, char newstorage, LOCKMODE lockmode) The comment should refer to "newcompression" not attcompression. Maybe you'd want to assert that exactly one of these is valid. Or maybe you should allow setting *both* if they're both valid. > case AMTYPE_TABLE: > return "TABLE"; >+ case AMTYPE_COMPRESSION: >+ return "TABLE"; Looks like this part got confused, maybe during rebase. > + if (strcmp(def->defname, "acceleration") == 0) > + { > + int32 acceleration = > + pg_atoi(defGetString(def), sizeof(acceleration), 0); > + > + if (acceleration < INT32_MIN || acceleration > INT32_MAX) Is this even reachable given that it's int32 type ? What do you think about this idea: On Sun, Jan 03, 2021 at 07:22:20PM -0600, Justin Pryzby wrote: > I think there should also be an option for pg_restore, like --no-tablespaces. > And I think there should be a GUC for default_compression, like > default_table_access_method, so one can restore into an alternate compression > by setting PGOPTIONS=-cdefault_compression=lz4. > > I'd like to be able to make all compressible columns of a table use a > non-default compression (except those which cannot), without having to use > \gexec... We have tables with up to 1600 columns. So a GUC would allow that. > > Previously (on separate threads) I wondered whether pg_dump > --no-table-access-method was needed - maybe that be sufficient for this case, > too, but I think it should be possible to separately avoid restoring > compression AM and AM "proper". So maybe it'd be like --no-tableam=compress > --no-tableam=storage or --no-tableam-all. I implemented the GUC. I'm sending a patchset rebased with this as 0002, and an un-rebased version in case you prefer that. I think the pg_dump half (--no-acccess-method=compress) is independently useful for tableams. > > Your first patch is large due to updating a large number of test cases to > > include the "compression" column in \d+ output. Maybe that column should be > > hidden when HIDE_TABLEAM is set by pg_regress ? I think that would allow > > testing with alternate, default compression. > > I am not sure whether we should hide the compression method when > HIDE_TABLEAM is set. I agree that it is actually an access method but > it is not the same right? Because we are using it for compression not > for storing data. You're right that TABLEAM should only be for table AM's. So I suggest to do something similar for HIDE_COMPRESSAM. That allows testing with an alternate, default compression (which is possible with the added GUC). Language/style fixen to follow: > + /* acoid should be found in some cases */ amoid > +-- test alter compression method for the partioned table partitioned > + /* > + * Loop for all the attributes in the tuple and check if any of the > + * attribute is compressed in the source tuple and its compression method > + * is not same as the target compression method then we need to decompress > + * it. Loop *over* any attribute is compressed (or any of the attributes *are* compressed) > + * If we have decompressed any of the field in the source tuple then free > + * the existing tuple. any field (or any of the fields) > + /* Likewise, copy compression if requested */ > + if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION > + && OidIsValid(attribute->attcompression)) Should say (.. & ..) != 0 && OidIsValid() > + else if (strcmp(def->compression, compression)) I think postgres likes you to explicitly write != 0 > + n->def = (Node *) makeString($5);; The 0002 patch has this double semicolon, which is removed by 0003. > + By default, each column in a partition inherits the compression method > + from its parent table, however a different compression method can be set > + for each partition. I guess it should say that it inherits from the parent table's *column*. > This clause specifies the type of access method to define. > - Only TABLE and INDEX > + TABLE, INDEX and COMPRESSION > are supported at present. I think "ONLY" got lost (?) Maybe it should say: "Currently, TABLE, INDEX, and COMPRESSION access methods are supported. -- Justin --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0001-Built-in-compression-method.patch" From 6aab69e3196e2e15b679050e2db2fc9907624715 Mon Sep 17 00:00:00 2001 From: dilipkumar Date: Tue, 8 Sep 2020 15:24:33 +0530 Subject: [PATCH v21 1/7] Built-in compression method Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com --- configure | 99 +++++++ configure.ac | 22 ++ contrib/test_decoding/expected/ddl.out | 50 ++-- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 ++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 72 +++-- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++-- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 147 ++++++++++ .../access/compression/compress_pglz.c | 131 +++++++++ .../access/compression/compressamapi.c | 61 +++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 6 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 26 +- src/backend/commands/createas.c | 7 + src/backend/commands/matview.c | 7 + src/backend/commands/tablecmds.c | 101 ++++++- src/backend/executor/nodeModifyTable.c | 83 ++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 8 + .../replication/logical/reorderbuffer.c | 1 + src/backend/utils/adt/pg_upgrade_support.c | 10 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 46 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 46 +++- src/bin/pg_dump/pg_dump.h | 2 + src/bin/psql/describe.c | 42 ++- src/include/access/compressamapi.h | 65 +++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 22 +- src/include/catalog/binary_upgrade.h | 2 + src/include/catalog/pg_am.dat | 7 +- src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 10 +- src/include/catalog/pg_proc.dat | 24 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 3 +- src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/alter_table.out | 10 +- src/test/regress/expected/compression.out | 176 ++++++++++++ src/test/regress/expected/compression_1.out | 171 ++++++++++++ src/test/regress/expected/copy2.out | 8 +- src/test/regress/expected/create_table.out | 142 +++++----- .../regress/expected/create_table_like.out | 88 +++--- src/test/regress/expected/domain.out | 16 +- src/test/regress/expected/foreign_data.out | 258 +++++++++--------- src/test/regress/expected/identity.out | 16 +- src/test/regress/expected/inherit.out | 138 +++++----- src/test/regress/expected/insert.out | 118 ++++---- src/test/regress/expected/matview.out | 80 +++--- src/test/regress/expected/psql.out | 108 ++++---- src/test/regress/expected/publication.out | 40 +-- .../regress/expected/replica_identity.out | 14 +- src/test/regress/expected/rowsecurity.out | 16 +- src/test/regress/expected/rules.out | 30 +- src/test/regress/expected/stats_ext.out | 10 +- src/test/regress/expected/update.out | 16 +- src/test/regress/output/tablespace.source | 16 +- src/test/regress/parallel_schedule | 3 + src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 74 +++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 81 files changed, 2193 insertions(+), 668 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/backend/access/compression/compressamapi.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index e202697bbf..a7b4317b90 100755 --- a/configure +++ b/configure @@ -698,6 +698,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -865,6 +866,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld enable_largefile ' @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --without-lz4 do not use lz4 --with-gnu-ld assume the C compiler uses GNU ld [default=no] Some influential environment variables: @@ -8596,6 +8599,35 @@ fi +# +# lz4 +# + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + : + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12087,6 +12119,59 @@ fi fi +if test "$with_lz4" = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "lz4 library not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support." "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13290,6 +13375,20 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + +else + as_fn_error $? "lz4 header not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support." "$LINENO" 5 +fi + + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index a5ad072ee4..ce29d6e97f 100644 --- a/configure.ac +++ b/configure.ac @@ -995,6 +995,13 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# lz4 +# +PGAC_ARG_BOOL(with, lz4, yes, + [do not use lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1182,6 +1189,14 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_LIB(lz4, LZ4_compress, [], + [AC_MSG_ERROR([lz4 library not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support.])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1397,6 +1412,13 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADER(lz4.h, [], [AC_MSG_ERROR([lz4 header not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support.])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..5bf614cc8a 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( PostgreSQL tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from its parent table, however a different compression method can be set + for each partition. diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..d514c7f8c1 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( sequence_options ) ] | UNIQUE index_parameters | PRIMARY KEY index_parameters | + COMPRESSION compression_method | REFERENCES reftable [ ( refcolumn ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE referential_action ] [ ON UPDATE referential_action ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS numeric_literal, REM + + INCLUDING COMPRESSION + + + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + + + + INCLUDING CONSTRAINTS @@ -981,6 +993,19 @@ WITH ( MODULUS numeric_literal, REM + + COMPRESSION compression_method + + + This clause adds the compression method to a column. The Compression + method can be set from available compression methods. The built-in + methods are pglz and lz4. + If no compression method is specified, then compressible types will have + the default compression method pglz. + + + + EXCLUDE [ USING index_method ] ( exclude_element WITH operator [, ... ] ) index_parameters [ WHERE ( predicate ) ] diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..c90464053c 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,47 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_handler - get the compression handler routines * - * Decompress a compressed version of a varlena datum + * helper function for toast_decompress_datum and toast_decompress_datum_slice */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) { - struct varlena *result; + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +512,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); - - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..c4814ef911 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o compressamapi.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..e8b035d138 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,147 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2016-2020, PostgreSQL Global Development Group + * Portions Copyright (c) 1990-1993, Regents of the University of California + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "postgres.h" +#include "access/compressamapi.h" +#include "access/toast_internals.h" +#include "fmgr.h" +#include "utils/builtins.h" + +#ifdef HAVE_LIBLZ4 +#include "lz4.h" +#endif + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + max_size = LZ4_compressBound(VARSIZE_ANY_EXHDR(value)); + tmp = (struct varlena *) palloc(max_size + TOAST_COMPRESS_HDRSZ); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + TOAST_COMPRESS_HDRSZ, + valsize, max_size); + if (len <= 0) + { + pfree(tmp); + elog(ERROR, "lz4: could not compress data"); + } + + SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = LZ4_decompress_safe(TOAST_COMPRESS_RAWDATA(value), + VARDATA(result), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + TOAST_COMPRESS_RAWSIZE(value)); + if (rawsize < 0) + elog(ERROR, "lz4: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = LZ4_decompress_safe_partial(TOAST_COMPRESS_RAWDATA(value), + VARDATA(result), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + slicelength, + TOAST_COMPRESS_RAWSIZE(value)); + if (rawsize < 0) + elog(ERROR, "lz4: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..578be6f1dd --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,131 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Copyright (c) 2015-2018, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/toast_internals.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + TOAST_COMPRESS_HDRSZ); + + len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), + valsize, + TOAST_COMPRESS_RAWDATA(tmp), + NULL); + + if (len >= 0) + { + SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + return tmp; + } + + pfree(tmp); + + return NULL; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), + TOAST_COMPRESS_SIZE(value), + VARDATA(result), + TOAST_COMPRESS_RAWSIZE(value), true); + + if (rawsize < 0) + elog(ERROR, "pglz: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + VARDATA(result), + slicelength, false); + + if (rawsize < 0) + elog(ERROR, "pglz: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c new file mode 100644 index 0000000000..663102c8d2 --- /dev/null +++ b/src/backend/access/compression/compressamapi.c @@ -0,0 +1,61 @@ +/*------------------------------------------------------------------------- + * + * compression/compressamapi.c + * Functions for compression methods + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/access/compression/compressamapi.c + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/htup.h" +#include "access/htup_details.h" +#include "access/reloptions.h" +#include "access/table.h" +#include "utils/fmgroids.h" +#include "utils/syscache.h" + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "Invalid compression method id %d", cmid); + } +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include #include +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index 009e215da1..44fa2ca7b4 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -181,6 +181,9 @@ my $C_COLLATION_OID = my $PG_CATALOG_NAMESPACE = Catalog::FindDefinedSymbolFromData($catalog_data{pg_namespace}, 'PG_CATALOG_NAMESPACE'); +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. @@ -808,6 +811,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b8cd35e995..e6040923dc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -347,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..668e43faed 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -34,6 +34,8 @@ static Oid lookup_am_handler_func(List *handler_name, char amtype); static const char *get_am_type_string(char amtype); +/* Set by pg_upgrade_support functions */ +Oid binary_upgrade_next_pg_am_oid = InvalidOid; /* * CreateAccessMethod @@ -83,7 +85,19 @@ CreateAccessMethod(CreateAmStmt *stmt) memset(values, 0, sizeof(values)); memset(nulls, false, sizeof(nulls)); - amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid); + if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid)) + { + /* acoid should be found in some cases */ + if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId && + (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid)) + elog(ERROR, "could not link to built-in attribute compression"); + + amoid = binary_upgrade_next_pg_am_oid; + binary_upgrade_next_pg_am_oid = InvalidOid; + } + else + amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid); + values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid); values[Anum_pg_am_amname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->amname)); @@ -175,6 +189,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..59690ce854 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -581,6 +581,13 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress(slot, myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..41c66c9e61 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -486,6 +486,13 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress(slot, myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..d7f4489c57 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,32 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * Get compression method Oid for the attribute. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + return InvalidOid; + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + return get_compression_am_oid(compression, false); +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 5d90337498..684e836071 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,12 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_internals.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2039,78 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. + */ +void +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool isnull = false; + bool decompressed_any = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return; + + /* + * Loop for all the attributes in the tuple and check if any of the + * attribute is compressed in the source tuple and its compression method + * is not same as the target compression method then we need to decompress + * it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value = (struct varlena *) + DatumGetPointer(slot_getattr(slot, attnum, &isnull)); + + /* nothing to be done, if the value is null */ + if (isnull) + continue; + + /* nothing to be done. if it is not compressed */ + if (!VARATT_IS_COMPRESSED(new_value)) + continue; + + /* + * Get the compression method stored in the toast header and + * compare with the compression method of the target. + */ + if (targetTupDesc->attrs[i].attcompression != + CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value))) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the field in the source tuple then free + * the existing tuple. + */ + if (decompressed_any) + { + /* deform complete tuple before we clear the existing tuple */ + slot_getallattrs(slot); + + ExecClearTuple(slot); + slot->tts_nvalid = natts; + slot->tts_flags &= ~TTS_FLAG_EMPTY; + } +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2319,14 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress( + slot, resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index ba3ccc712c..3ea32323cd 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2932,6 +2932,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index a2ef853dc2..227bb07bbd 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2598,6 +2598,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6be19916fc..c6ba9e1b05 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3858,6 +3858,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 8392be6d44..187d3570e5 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2865,6 +2865,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 7574d545e0..355e273192 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -595,6 +595,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type hash_partbound %type hash_partbound_elem +%type optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -630,9 +632,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3420,11 +3422,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3433,8 +3436,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3479,6 +3482,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3709,6 +3720,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15237,6 +15249,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15755,6 +15768,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..e2ac159aa2 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 5a62ab8bbc..43eee8a4b7 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -87,6 +87,7 @@ #include "access/detoast.h" #include "access/heapam.h" #include "access/rewriteheap.h" +#include "access/toast_internals.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog_internal.h" diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index a575c95079..f026104c01 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +Datum +binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_am_oid = amoid; + PG_RETURN_VOID(); +} + Datum binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS) { diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..aff5b7e95e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include #include +#include "access/toast_internals.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,50 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + char *compression; + int typlen; + struct varlena *varvalue; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + /* + * If it is not a varlena type or the attribute is not compressed then + * return NULL. + */ + if ((typlen != -1) || !VARATT_IS_COMPRESSED(value)) + PG_RETURN_NULL(); + + varvalue = (struct varlena *) DatumGetPointer(value); + + compression = + get_am_name(CompressionIdToOid(TOAST_COMPRESS_METHOD(varvalue))); + + PG_RETURN_TEXT_P(cstring_to_text(compression)); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 9d0056a569..5e168a3c04 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 39da742e32..becf565cdd 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -13016,6 +13035,11 @@ dumpAccessMethod(Archive *fout, AccessMethodInfo *aminfo) qamname = pg_strdup(fmtId(aminfo->dobj.name)); + if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION) + appendPQExpBuffer(q, + "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n", + aminfo->dobj.catId.oid); + appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname); switch (aminfo->amtype) @@ -15805,6 +15829,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15829,6 +15854,9 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15864,6 +15892,22 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Compression + * + * In binary-upgrade mode, compression is assigned by + * ALTER. Even if we're skipping compression the attribute + * will get default compression. It's the task for ALTER + * command to restore compression info. + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + has_non_default_compression) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 1290f9659b..53ece7abd7 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -327,6 +327,8 @@ typedef struct _tableInfo bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ + /* * Stuff computed only for dumpable tables. */ diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..4f82929fdb 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,27 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + bool mustfree = false; + const int trunclen = 100; + char *val = PQgetvalue(res, i, attcompression_col); + + /* truncate the options if they're too long */ + if (strlen(val) > trunclen + 3) + { + char *trunc = pg_malloc0(trunclen + 4); + strncpy(trunc, val, trunclen); + strncpy(trunc + trunclen, "...", 4); + + val = trunc; + mustfree = true; + } + + printTableAddCell(&cont, val, false, mustfree); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..8226ae0596 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,65 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Copyright (c) 2015-2017, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* access/compression/compressamapi.c */ +extern CompressionId CompressionOidToId(Oid cmoid); +extern Oid CompressionIdToOid(CompressionId cmid); + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..84ba303e7e 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,33 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; +#define RAWSIZEMASK (0x3FFFFFFFU) + /* * Utilities for manipulation of header information for compressed * toast entries. + * + * Since version 14 TOAST_COMPRESS_SET_RAWSIZE also marks compressed + * varlenas as custom compressed. Such varlenas will contain 0x02 (0b10) in + * two highest bits. */ #define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) +#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->info & RAWSIZEMASK) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> 30) #define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) #define TOAST_COMPRESS_RAWDATA(ptr) \ (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= RAWSIZEMASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << 30); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7ac5..9d65bae238 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid; + extern PGDLLIMPORT bool binary_upgrade_record_init_privs; #endif /* BINARY_UPGRADE_H */ diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..00362d9db3 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,10 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, - +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index 358f5aac8f..4a5c249ee9 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX(pg_am_oid_index, 2652, on pg_am using btree(oid oid_ops)); */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 059dec3647..a6cb1b4e5b 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -156,6 +156,14 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation */ Oid attcollation; + /* + * Oid of the compression method that will be used for compressing the value + * for this attribute. For compressible types this must always be a + * valid Oid irrespective of what is the current value of the attstorage. + * And for the incompressible types this must be InvalidOid. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -183,7 +191,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b5f52d4e4a..7458ed7d0d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7087,6 +7096,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7257,6 +7270,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, @@ -10749,6 +10769,10 @@ proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v', proparallel => 'r', prorettype => 'void', proargtypes => 'oid', prosrc => 'binary_upgrade_set_next_pg_authid_oid' }, +{ oid => '1137', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v', + proparallel => 'r', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_am_oid' }, { oid => '3591', descr => 'for use by pg_upgrade', proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f', provolatile => 'v', proparallel => 'u', prorettype => 'void', diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 56da2913bd..984002ba21 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -625,6 +625,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 758c3ca097..36d32f94aa 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -616,5 +616,6 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern void CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index caed683ba9..ded2ad9692 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -510,6 +510,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index dc2bb40926..232c9f1248 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 8c554e1f69..89bd2fd7a8 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -87,6 +87,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index f4d9f3b408..47f114b8a0 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..90c7454d2a 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -55,9 +55,9 @@ /* * struct varatt_external is a traditional "TOAST pointer", that is, the * information needed to fetch a Datum stored out-of-line in a TOAST table. - * The data is compressed if and only if va_extsize < va_rawsize - VARHDRSZ. - * This struct must not contain any padding, because we sometimes compare - * these pointers using memcmp. + * The data is compressed if and only if size in + * va_extinfo < va_rawsize - VARHDRSZ. This struct must not contain any + * padding, because we sometimes compare these pointers using memcmp. * * Note that this information is stored unaligned within actual tuples, so * you need to memcpy from the tuple into a local struct variable before @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -280,8 +281,11 @@ typedef struct #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & 0x3FFFFFFF) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> 30) /* Externally visible macros */ diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 0ce6ee4622..138df4be93 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -2196,11 +2196,11 @@ where oid = 'test_storage'::regclass; create index test_storage_idx on test_storage (b, a); alter table test_storage alter column a set storage external; \d+ test_storage - Table "public.test_storage" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | external | | - b | integer | | | 0 | plain | | + Table "public.test_storage" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | external | pglz | | + b | integer | | | 0 | plain | | | Indexes: "test_storage_idx" btree (b, a) diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..4dea269fc3 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,176 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..a50c7679bb --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,171 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: not built with lz4 support +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +-------- +(0 rows) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- +(0 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: not built with lz4 support +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- +(0 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out index c64f0719e7..1778c15c7a 100644 --- a/src/test/regress/expected/copy2.out +++ b/src/test/regress/expected/copy2.out @@ -511,10 +511,10 @@ begin end $$ language plpgsql immutable; alter table check_con_tbl add check (check_con_function(check_con_tbl.*)); \d+ check_con_tbl - Table "public.check_con_tbl" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | + Table "public.check_con_tbl" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | Check constraints: "check_con_tbl_check" CHECK (check_con_function(check_con_tbl.*)) diff --git a/src/test/regress/expected/create_table.out b/src/test/regress/expected/create_table.out index ed8c01b8de..3105115fee 100644 --- a/src/test/regress/expected/create_table.out +++ b/src/test/regress/expected/create_table.out @@ -498,11 +498,11 @@ Partition key: RANGE (a oid_ops, plusone(b), c, d COLLATE "C") Number of partitions: 0 \d+ partitioned2 - Partitioned table "public.partitioned2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | integer | | | | plain | | - b | text | | | | extended | | + Partitioned table "public.partitioned2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | text | | | | extended | pglz | | Partition key: RANGE (((a + 1)), substr(b, 1, 5)) Number of partitions: 0 @@ -511,11 +511,11 @@ ERROR: no partition of relation "partitioned2" found for row DETAIL: Partition key of the failing row contains ((a + 1), substr(b, 1, 5)) = (2, hello). CREATE TABLE part2_1 PARTITION OF partitioned2 FOR VALUES FROM (-1, 'aaaaa') TO (100, 'ccccc'); \d+ part2_1 - Table "public.part2_1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | integer | | | | plain | | - b | text | | | | extended | | + Table "public.part2_1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | text | | | | extended | pglz | | Partition of: partitioned2 FOR VALUES FROM ('-1', 'aaaaa') TO (100, 'ccccc') Partition constraint: (((a + 1) IS NOT NULL) AND (substr(b, 1, 5) IS NOT NULL) AND (((a + 1) > '-1'::integer) OR (((a + 1) = '-1'::integer) AND (substr(b, 1, 5) >= 'aaaaa'::text))) AND (((a + 1) < 100) OR (((a + 1) = 100) AND (substr(b, 1, 5) < 'ccccc'::text)))) @@ -552,11 +552,11 @@ select * from partitioned where partitioned = '(1,2)'::partitioned; (2 rows) \d+ partitioned1 - Table "public.partitioned1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | + Table "public.partitioned1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | Partition of: partitioned FOR VALUES IN ('(1,2)') Partition constraint: (((partitioned1.*)::partitioned IS DISTINCT FROM NULL) AND ((partitioned1.*)::partitioned = '(1,2)'::partitioned)) @@ -609,10 +609,10 @@ CREATE TABLE part_p2 PARTITION OF list_parted FOR VALUES IN (2); CREATE TABLE part_p3 PARTITION OF list_parted FOR VALUES IN ((2+1)); CREATE TABLE part_null PARTITION OF list_parted FOR VALUES IN (null); \d+ list_parted - Partitioned table "public.list_parted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Partitioned table "public.list_parted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition key: LIST (a) Partitions: part_null FOR VALUES IN (NULL), part_p1 FOR VALUES IN (1), @@ -1049,21 +1049,21 @@ create table test_part_coll_cast2 partition of test_part_coll_posix for values f drop table test_part_coll_posix; -- Partition bound in describe output \d+ part_b - Table "public.part_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 1 | plain | | + Table "public.part_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 1 | plain | | | Partition of: parted FOR VALUES IN ('b') Partition constraint: ((a IS NOT NULL) AND (a = 'b'::text)) -- Both partition bound and partition key in describe output \d+ part_c - Partitioned table "public.part_c" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 0 | plain | | + Partitioned table "public.part_c" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 0 | plain | | | Partition of: parted FOR VALUES IN ('c') Partition constraint: ((a IS NOT NULL) AND (a = 'c'::text)) Partition key: RANGE (b) @@ -1071,11 +1071,11 @@ Partitions: part_c_1_10 FOR VALUES FROM (1) TO (10) -- a level-2 partition's constraint will include the parent's expressions \d+ part_c_1_10 - Table "public.part_c_1_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 0 | plain | | + Table "public.part_c_1_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 0 | plain | | | Partition of: part_c FOR VALUES FROM (1) TO (10) Partition constraint: ((a IS NOT NULL) AND (a = 'c'::text) AND (b IS NOT NULL) AND (b >= 1) AND (b < 10)) @@ -1104,46 +1104,46 @@ Number of partitions: 3 (Use \d+ to list them.) CREATE TABLE range_parted4 (a int, b int, c int) PARTITION BY RANGE (abs(a), abs(b), c); CREATE TABLE unbounded_range_part PARTITION OF range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (MAXVALUE, MAXVALUE, MAXVALUE); \d+ unbounded_range_part - Table "public.unbounded_range_part" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.unbounded_range_part" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (MAXVALUE, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL)) DROP TABLE unbounded_range_part; CREATE TABLE range_parted4_1 PARTITION OF range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (1, MAXVALUE, MAXVALUE); \d+ range_parted4_1 - Table "public.range_parted4_1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (1, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND (abs(a) <= 1)) CREATE TABLE range_parted4_2 PARTITION OF range_parted4 FOR VALUES FROM (3, 4, 5) TO (6, 7, MAXVALUE); \d+ range_parted4_2 - Table "public.range_parted4_2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (3, 4, 5) TO (6, 7, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND ((abs(a) > 3) OR ((abs(a) = 3) AND (abs(b) > 4)) OR ((abs(a) = 3) AND (abs(b) = 4) AND (c >= 5))) AND ((abs(a) < 6) OR ((abs(a) = 6) AND (abs(b) <= 7)))) CREATE TABLE range_parted4_3 PARTITION OF range_parted4 FOR VALUES FROM (6, 8, MINVALUE) TO (9, MAXVALUE, MAXVALUE); \d+ range_parted4_3 - Table "public.range_parted4_3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (6, 8, MINVALUE) TO (9, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND ((abs(a) > 6) OR ((abs(a) = 6) AND (abs(b) >= 8))) AND (abs(a) <= 9)) @@ -1175,11 +1175,11 @@ SELECT obj_description('parted_col_comment'::regclass); (1 row) \d+ parted_col_comment - Partitioned table "public.parted_col_comment" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+--------------- - a | integer | | | | plain | | Partition key - b | text | | | | extended | | + Partitioned table "public.parted_col_comment" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+--------------- + a | integer | | | | plain | | | Partition key + b | text | | | | extended | pglz | | Partition key: LIST (a) Number of partitions: 0 @@ -1188,10 +1188,10 @@ DROP TABLE parted_col_comment; CREATE TABLE arrlp (a int[]) PARTITION BY LIST (a); CREATE TABLE arrlp12 PARTITION OF arrlp FOR VALUES IN ('{1}', '{2}'); \d+ arrlp12 - Table "public.arrlp12" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-----------+-----------+----------+---------+----------+--------------+------------- - a | integer[] | | | | extended | | + Table "public.arrlp12" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-----------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer[] | | | | extended | pglz | | Partition of: arrlp FOR VALUES IN ('{1}', '{2}') Partition constraint: ((a IS NOT NULL) AND ((a = '{1}'::integer[]) OR (a = '{2}'::integer[]))) @@ -1201,10 +1201,10 @@ create table boolspart (a bool) partition by list (a); create table boolspart_t partition of boolspart for values in (true); create table boolspart_f partition of boolspart for values in (false); \d+ boolspart - Partitioned table "public.boolspart" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | boolean | | | | plain | | + Partitioned table "public.boolspart" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | boolean | | | | plain | | | Partition key: LIST (a) Partitions: boolspart_f FOR VALUES IN (false), boolspart_t FOR VALUES IN (true) diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out index 10d17be23c..4da878ee1c 100644 --- a/src/test/regress/expected/create_table_like.out +++ b/src/test/regress/expected/create_table_like.out @@ -325,32 +325,32 @@ CREATE TABLE ctlt4 (a text, c text); ALTER TABLE ctlt4 ALTER COLUMN c SET STORAGE EXTERNAL; CREATE TABLE ctlt12_storage (LIKE ctlt1 INCLUDING STORAGE, LIKE ctlt2 INCLUDING STORAGE); \d+ ctlt12_storage - Table "public.ctlt12_storage" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | - b | text | | | | extended | | - c | text | | | | external | | + Table "public.ctlt12_storage" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | CREATE TABLE ctlt12_comments (LIKE ctlt1 INCLUDING COMMENTS, LIKE ctlt2 INCLUDING COMMENTS); \d+ ctlt12_comments - Table "public.ctlt12_comments" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | extended | | A - b | text | | | | extended | | B - c | text | | | | extended | | C + Table "public.ctlt12_comments" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | extended | pglz | | A + b | text | | | | extended | pglz | | B + c | text | | | | extended | pglz | | C CREATE TABLE ctlt1_inh (LIKE ctlt1 INCLUDING CONSTRAINTS INCLUDING COMMENTS) INHERITS (ctlt1); NOTICE: merging column "a" with inherited definition NOTICE: merging column "b" with inherited definition NOTICE: merging constraint "ctlt1_a_check" with inherited definition \d+ ctlt1_inh - Table "public.ctlt1_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.ctlt1_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Inherits: ctlt1 @@ -364,12 +364,12 @@ SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_con CREATE TABLE ctlt13_inh () INHERITS (ctlt1, ctlt3); NOTICE: merging multiple inherited definitions of column "a" \d+ ctlt13_inh - Table "public.ctlt13_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | - b | text | | | | extended | | - c | text | | | | external | | + Table "public.ctlt13_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) "ctlt3_a_check" CHECK (length(a) < 5) @@ -380,12 +380,12 @@ Inherits: ctlt1, CREATE TABLE ctlt13_like (LIKE ctlt3 INCLUDING CONSTRAINTS INCLUDING INDEXES INCLUDING COMMENTS INCLUDING STORAGE) INHERITS (ctlt1); NOTICE: merging column "a" with inherited definition \d+ ctlt13_like - Table "public.ctlt13_like" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A3 - b | text | | | | extended | | - c | text | | | | external | | C + Table "public.ctlt13_like" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A3 + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | C Indexes: "ctlt13_like_expr_idx" btree ((a || c)) Check constraints: @@ -402,11 +402,11 @@ SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_con CREATE TABLE ctlt_all (LIKE ctlt1 INCLUDING ALL); \d+ ctlt_all - Table "public.ctlt_all" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.ctlt_all" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "ctlt_all_pkey" PRIMARY KEY, btree (a) "ctlt_all_b_idx" btree (b) @@ -440,11 +440,11 @@ DETAIL: MAIN versus EXTENDED -- Check that LIKE isn't confused by a system catalog of the same name CREATE TABLE pg_attrdef (LIKE ctlt1 INCLUDING ALL); \d+ public.pg_attrdef - Table "public.pg_attrdef" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.pg_attrdef" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "pg_attrdef_pkey" PRIMARY KEY, btree (a) "pg_attrdef_b_idx" btree (b) @@ -461,11 +461,11 @@ CREATE SCHEMA ctl_schema; SET LOCAL search_path = ctl_schema, public; CREATE TABLE ctlt1 (LIKE ctlt1 INCLUDING ALL); \d+ ctlt1 - Table "ctl_schema.ctlt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "ctl_schema.ctlt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "ctlt1_pkey" PRIMARY KEY, btree (a) "ctlt1_b_idx" btree (b) diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out index 411d5c003e..6268b26562 100644 --- a/src/test/regress/expected/domain.out +++ b/src/test/regress/expected/domain.out @@ -266,10 +266,10 @@ explain (verbose, costs off) create rule silly as on delete to dcomptable do instead update dcomptable set d1.r = (d1).r - 1, d1.i = (d1).i + 1 where (d1).i > 0; \d+ dcomptable - Table "public.dcomptable" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-----------+-----------+----------+---------+----------+--------------+------------- - d1 | dcomptype | | | | extended | | + Table "public.dcomptable" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-----------+-----------+----------+---------+----------+-------------+--------------+------------- + d1 | dcomptype | | | | extended | pglz | | Indexes: "dcomptable_d1_key" UNIQUE CONSTRAINT, btree (d1) Rules: @@ -403,10 +403,10 @@ create rule silly as on delete to dcomptable do instead update dcomptable set d1[1].r = d1[1].r - 1, d1[1].i = d1[1].i + 1 where d1[1].i > 0; \d+ dcomptable - Table "public.dcomptable" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------------+-----------+----------+---------+----------+--------------+------------- - d1 | dcomptypea | | | | extended | | + Table "public.dcomptable" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------------+-----------+----------+---------+----------+-------------+--------------+------------- + d1 | dcomptypea | | | | extended | pglz | | Indexes: "dcomptable_d1_key" UNIQUE CONSTRAINT, btree (d1) Rules: diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out index b9e25820bc..89466da603 100644 --- a/src/test/regress/expected/foreign_data.out +++ b/src/test/regress/expected/foreign_data.out @@ -1383,12 +1383,12 @@ CREATE TABLE fd_pt1 ( CREATE FOREIGN TABLE ft2 () INHERITS (fd_pt1) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value'); \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1404,12 +1404,12 @@ Inherits: fd_pt1 DROP FOREIGN TABLE ft2; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | CREATE FOREIGN TABLE ft2 ( c1 integer NOT NULL, @@ -1428,12 +1428,12 @@ FDW options: (delimiter ',', quote '"', "be quoted" 'value') ALTER FOREIGN TABLE ft2 INHERIT fd_pt1; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1471,12 +1471,12 @@ Child tables: ct3, ft3 \d+ ct3 - Table "public.ct3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.ct3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Inherits: ft2 \d+ ft3 @@ -1496,17 +1496,17 @@ ALTER TABLE fd_pt1 ADD COLUMN c6 integer; ALTER TABLE fd_pt1 ADD COLUMN c7 integer NOT NULL; ALTER TABLE fd_pt1 ADD COLUMN c8 integer; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | | plain | | - c5 | integer | | | 0 | plain | | - c6 | integer | | | | plain | | - c7 | integer | | not null | | plain | | - c8 | integer | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | | plain | | | + c5 | integer | | | 0 | plain | | | + c6 | integer | | | | plain | | | + c7 | integer | | not null | | plain | | | + c8 | integer | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1528,17 +1528,17 @@ Child tables: ct3, ft3 \d+ ct3 - Table "public.ct3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | | plain | | - c5 | integer | | | 0 | plain | | - c6 | integer | | | | plain | | - c7 | integer | | not null | | plain | | - c8 | integer | | | | plain | | + Table "public.ct3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | | plain | | | + c5 | integer | | | 0 | plain | | | + c6 | integer | | | | plain | | | + c7 | integer | | not null | | plain | | | + c8 | integer | | | | plain | | | Inherits: ft2 \d+ ft3 @@ -1570,17 +1570,17 @@ ALTER TABLE fd_pt1 ALTER COLUMN c1 SET (n_distinct = 100); ALTER TABLE fd_pt1 ALTER COLUMN c8 SET STATISTICS -1; ALTER TABLE fd_pt1 ALTER COLUMN c8 SET STORAGE EXTERNAL; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | 0 | plain | | - c5 | integer | | | | plain | | - c6 | integer | | not null | | plain | | - c7 | integer | | | | plain | | - c8 | text | | | | external | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | 0 | plain | | | + c5 | integer | | | | plain | | | + c6 | integer | | not null | | plain | | | + c7 | integer | | | | plain | | | + c8 | text | | | | external | pglz | | Child tables: ft2 \d+ ft2 @@ -1608,12 +1608,12 @@ ALTER TABLE fd_pt1 DROP COLUMN c6; ALTER TABLE fd_pt1 DROP COLUMN c7; ALTER TABLE fd_pt1 DROP COLUMN c8; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1645,12 +1645,12 @@ SELECT relname, conname, contype, conislocal, coninhcount, connoinherit -- child does not inherit NO INHERIT constraints \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk1" CHECK (c1 > 0) NO INHERIT "fd_pt1chk2" CHECK (c2 <> ''::text) @@ -1692,12 +1692,12 @@ ALTER FOREIGN TABLE ft2 ADD CONSTRAINT fd_pt1chk2 CHECK (c2 <> ''); ALTER FOREIGN TABLE ft2 INHERIT fd_pt1; -- child does not inherit NO INHERIT constraints \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk1" CHECK (c1 > 0) NO INHERIT "fd_pt1chk2" CHECK (c2 <> ''::text) @@ -1723,12 +1723,12 @@ ALTER TABLE fd_pt1 DROP CONSTRAINT fd_pt1chk2 CASCADE; INSERT INTO fd_pt1 VALUES (1, 'fd_pt1'::text, '1994-01-01'::date); ALTER TABLE fd_pt1 ADD CONSTRAINT fd_pt1chk3 CHECK (c2 <> '') NOT VALID; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk3" CHECK (c2 <> ''::text) NOT VALID Child tables: ft2 @@ -1750,12 +1750,12 @@ Inherits: fd_pt1 -- VALIDATE CONSTRAINT need do nothing on foreign tables ALTER TABLE fd_pt1 VALIDATE CONSTRAINT fd_pt1chk3; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk3" CHECK (c2 <> ''::text) Child tables: ft2 @@ -1781,12 +1781,12 @@ ALTER TABLE fd_pt1 RENAME COLUMN c3 TO f3; -- changes name of a constraint recursively ALTER TABLE fd_pt1 RENAME CONSTRAINT fd_pt1chk3 TO f2_check; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | not null | | plain | 10000 | - f2 | text | | | | extended | | - f3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | not null | | plain | | 10000 | + f2 | text | | | | extended | pglz | | + f3 | date | | | | plain | | | Check constraints: "f2_check" CHECK (f2 <> ''::text) Child tables: ft2 @@ -1845,12 +1845,12 @@ CREATE TABLE fd_pt2 ( CREATE FOREIGN TABLE fd_pt2_1 PARTITION OF fd_pt2 FOR VALUES IN (1) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value'); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1890,12 +1890,12 @@ ERROR: table "fd_pt2_1" contains column "c4" not found in parent "fd_pt2" DETAIL: The new partition may contain only the columns present in parent. DROP FOREIGN TABLE fd_pt2_1; \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Number of partitions: 0 @@ -1917,12 +1917,12 @@ FDW options: (delimiter ',', quote '"', "be quoted" 'value') -- no attach partition validation occurs for foreign tables ALTER TABLE fd_pt2 ATTACH PARTITION fd_pt2_1 FOR VALUES IN (1); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1945,12 +1945,12 @@ ERROR: cannot add column to a partition ALTER TABLE fd_pt2_1 ALTER c3 SET NOT NULL; ALTER TABLE fd_pt2_1 ADD CONSTRAINT p21chk CHECK (c2 <> ''); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1975,12 +1975,12 @@ ERROR: column "c1" is marked NOT NULL in parent table ALTER TABLE fd_pt2 DETACH PARTITION fd_pt2_1; ALTER TABLE fd_pt2 ALTER c2 SET NOT NULL; \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | not null | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | not null | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Number of partitions: 0 @@ -2003,12 +2003,12 @@ ALTER TABLE fd_pt2 ATTACH PARTITION fd_pt2_1 FOR VALUES IN (1); ALTER TABLE fd_pt2 DETACH PARTITION fd_pt2_1; ALTER TABLE fd_pt2 ADD CONSTRAINT fd_pt2chk1 CHECK (c1 > 0); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | not null | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | not null | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Check constraints: "fd_pt2chk1" CHECK (c1 > 0) diff --git a/src/test/regress/expected/identity.out b/src/test/regress/expected/identity.out index fbca0333a2..dc2af075ff 100644 --- a/src/test/regress/expected/identity.out +++ b/src/test/regress/expected/identity.out @@ -498,14 +498,14 @@ TABLE itest8; (2 rows) \d+ itest8 - Table "public.itest8" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+----------------------------------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | not null | generated always as identity | plain | | - f3 | integer | | not null | generated by default as identity | plain | | - f4 | bigint | | not null | generated always as identity | plain | | - f5 | bigint | | | | plain | | + Table "public.itest8" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+----------------------------------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | not null | generated always as identity | plain | | | + f3 | integer | | not null | generated by default as identity | plain | | | + f4 | bigint | | not null | generated always as identity | plain | | | + f5 | bigint | | | | plain | | | \d itest8_f2_seq Sequence "public.itest8_f2_seq" diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out index 2b68aef654..7e709916f6 100644 --- a/src/test/regress/expected/inherit.out +++ b/src/test/regress/expected/inherit.out @@ -1052,13 +1052,13 @@ ALTER TABLE inhts RENAME aa TO aaa; -- to be failed ERROR: cannot rename inherited column "aa" ALTER TABLE inhts RENAME d TO dd; \d+ inhts - Table "public.inhts" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aa | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | - dd | integer | | | | plain | | + Table "public.inhts" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aa | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | + dd | integer | | | | plain | | | Inherits: inht1, inhs1 @@ -1071,14 +1071,14 @@ NOTICE: merging multiple inherited definitions of column "aa" NOTICE: merging multiple inherited definitions of column "b" ALTER TABLE inht1 RENAME aa TO aaa; \d+ inht4 - Table "public.inht4" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aaa | integer | | | | plain | | - b | integer | | | | plain | | - x | integer | | | | plain | | - y | integer | | | | plain | | - z | integer | | | | plain | | + Table "public.inht4" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aaa | integer | | | | plain | | | + b | integer | | | | plain | | | + x | integer | | | | plain | | | + y | integer | | | | plain | | | + z | integer | | | | plain | | | Inherits: inht2, inht3 @@ -1088,14 +1088,14 @@ ALTER TABLE inht1 RENAME aaa TO aaaa; ALTER TABLE inht1 RENAME b TO bb; -- to be failed ERROR: cannot rename inherited column "b" \d+ inhts - Table "public.inhts" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aaaa | integer | | | | plain | | - b | integer | | | | plain | | - x | integer | | | | plain | | - c | integer | | | | plain | | - d | integer | | | | plain | | + Table "public.inhts" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aaaa | integer | | | | plain | | | + b | integer | | | | plain | | | + x | integer | | | | plain | | | + c | integer | | | | plain | | | + d | integer | | | | plain | | | Inherits: inht2, inhs1 @@ -1135,33 +1135,33 @@ drop cascades to table inht4 CREATE TABLE test_constraints (id int, val1 varchar, val2 int, UNIQUE(val1, val2)); CREATE TABLE test_constraints_inh () INHERITS (test_constraints); \d+ test_constraints - Table "public.test_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Indexes: "test_constraints_val1_val2_key" UNIQUE CONSTRAINT, btree (val1, val2) Child tables: test_constraints_inh ALTER TABLE ONLY test_constraints DROP CONSTRAINT test_constraints_val1_val2_key; \d+ test_constraints - Table "public.test_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Child tables: test_constraints_inh \d+ test_constraints_inh - Table "public.test_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Inherits: test_constraints DROP TABLE test_constraints_inh; @@ -1172,27 +1172,27 @@ CREATE TABLE test_ex_constraints ( ); CREATE TABLE test_ex_constraints_inh () INHERITS (test_ex_constraints); \d+ test_ex_constraints - Table "public.test_ex_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Indexes: "test_ex_constraints_c_excl" EXCLUDE USING gist (c WITH &&) Child tables: test_ex_constraints_inh ALTER TABLE test_ex_constraints DROP CONSTRAINT test_ex_constraints_c_excl; \d+ test_ex_constraints - Table "public.test_ex_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Child tables: test_ex_constraints_inh \d+ test_ex_constraints_inh - Table "public.test_ex_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Inherits: test_ex_constraints DROP TABLE test_ex_constraints_inh; @@ -1202,37 +1202,37 @@ CREATE TABLE test_primary_constraints(id int PRIMARY KEY); CREATE TABLE test_foreign_constraints(id1 int REFERENCES test_primary_constraints(id)); CREATE TABLE test_foreign_constraints_inh () INHERITS (test_foreign_constraints); \d+ test_primary_constraints - Table "public.test_primary_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id | integer | | not null | | plain | | + Table "public.test_primary_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id | integer | | not null | | plain | | | Indexes: "test_primary_constraints_pkey" PRIMARY KEY, btree (id) Referenced by: TABLE "test_foreign_constraints" CONSTRAINT "test_foreign_constraints_id1_fkey" FOREIGN KEY (id1) REFERENCES test_primary_constraints(id) \d+ test_foreign_constraints - Table "public.test_foreign_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Foreign-key constraints: "test_foreign_constraints_id1_fkey" FOREIGN KEY (id1) REFERENCES test_primary_constraints(id) Child tables: test_foreign_constraints_inh ALTER TABLE test_foreign_constraints DROP CONSTRAINT test_foreign_constraints_id1_fkey; \d+ test_foreign_constraints - Table "public.test_foreign_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Child tables: test_foreign_constraints_inh \d+ test_foreign_constraints_inh - Table "public.test_foreign_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Inherits: test_foreign_constraints DROP TABLE test_foreign_constraints_inh; diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index da50ee3b67..3cd4ad9e53 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -142,11 +142,11 @@ create rule irule3 as on insert to inserttest2 do also insert into inserttest (f4[1].if1, f4[1].if2[2]) select new.f1, new.f2; \d+ inserttest2 - Table "public.inserttest2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+----------+--------------+------------- - f1 | bigint | | | | plain | | - f2 | text | | | | extended | | + Table "public.inserttest2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | bigint | | | | plain | | | + f2 | text | | | | extended | pglz | | Rules: irule1 AS ON INSERT TO inserttest2 DO INSERT INTO inserttest (f3.if2[1], f3.if2[2]) @@ -448,11 +448,11 @@ from hash_parted order by part; -- test \d+ output on a table which has both partitioned and unpartitioned -- partitions \d+ list_parted - Partitioned table "public.list_parted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Partitioned table "public.list_parted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition key: LIST (lower(a)) Partitions: part_aa_bb FOR VALUES IN ('aa', 'bb'), part_cc_dd FOR VALUES IN ('cc', 'dd'), @@ -470,10 +470,10 @@ drop table hash_parted; create table list_parted (a int) partition by list (a); create table part_default partition of list_parted default; \d+ part_default - Table "public.part_default" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Table "public.part_default" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition of: list_parted DEFAULT No partition constraint @@ -853,11 +853,11 @@ create table mcrparted6_common_ge_10 partition of mcrparted for values from ('co create table mcrparted7_gt_common_lt_d partition of mcrparted for values from ('common', maxvalue) to ('d', minvalue); create table mcrparted8_ge_d partition of mcrparted for values from ('d', minvalue) to (maxvalue, maxvalue); \d+ mcrparted - Partitioned table "public.mcrparted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Partitioned table "public.mcrparted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition key: RANGE (a, b) Partitions: mcrparted1_lt_b FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVALUE), mcrparted2_b FOR VALUES FROM ('b', MINVALUE) TO ('c', MINVALUE), @@ -869,74 +869,74 @@ Partitions: mcrparted1_lt_b FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVAL mcrparted8_ge_d FOR VALUES FROM ('d', MINVALUE) TO (MAXVALUE, MAXVALUE) \d+ mcrparted1_lt_b - Table "public.mcrparted1_lt_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted1_lt_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a < 'b'::text)) \d+ mcrparted2_b - Table "public.mcrparted2_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted2_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('b', MINVALUE) TO ('c', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'b'::text) AND (a < 'c'::text)) \d+ mcrparted3_c_to_common - Table "public.mcrparted3_c_to_common" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted3_c_to_common" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('c', MINVALUE) TO ('common', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'c'::text) AND (a < 'common'::text)) \d+ mcrparted4_common_lt_0 - Table "public.mcrparted4_common_lt_0" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted4_common_lt_0" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', MINVALUE) TO ('common', 0) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b < 0)) \d+ mcrparted5_common_0_to_10 - Table "public.mcrparted5_common_0_to_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted5_common_0_to_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', 0) TO ('common', 10) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b >= 0) AND (b < 10)) \d+ mcrparted6_common_ge_10 - Table "public.mcrparted6_common_ge_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted6_common_ge_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', 10) TO ('common', MAXVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b >= 10)) \d+ mcrparted7_gt_common_lt_d - Table "public.mcrparted7_gt_common_lt_d" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted7_gt_common_lt_d" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', MAXVALUE) TO ('d', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a > 'common'::text) AND (a < 'd'::text)) \d+ mcrparted8_ge_d - Table "public.mcrparted8_ge_d" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted8_ge_d" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('d', MINVALUE) TO (MAXVALUE, MAXVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'd'::text)) diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out index 4b3a2e0cb7..e078818496 100644 --- a/src/test/regress/expected/matview.out +++ b/src/test/regress/expected/matview.out @@ -94,11 +94,11 @@ CREATE MATERIALIZED VIEW mvtest_bb AS SELECT * FROM mvtest_tvvmv; CREATE INDEX mvtest_aa ON mvtest_bb (grandtot); -- check that plans seem reasonable \d+ mvtest_tvm - Materialized view "public.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "public.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -106,11 +106,11 @@ View definition: ORDER BY mvtest_tv.type; \d+ mvtest_tvm - Materialized view "public.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "public.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -118,19 +118,19 @@ View definition: ORDER BY mvtest_tv.type; \d+ mvtest_tvvm - Materialized view "public.mvtest_tvvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_tvvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | View definition: SELECT mvtest_tvv.grandtot FROM mvtest_tvv; \d+ mvtest_bb - Materialized view "public.mvtest_bb" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_bb" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | Indexes: "mvtest_aa" btree (grandtot) View definition: @@ -142,10 +142,10 @@ CREATE SCHEMA mvtest_mvschema; ALTER MATERIALIZED VIEW mvtest_tvm SET SCHEMA mvtest_mvschema; \d+ mvtest_tvm \d+ mvtest_tvmm - Materialized view "public.mvtest_tvmm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_tvmm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | Indexes: "mvtest_tvmm_expr" UNIQUE, btree ((grandtot > 0::numeric)) "mvtest_tvmm_pred" UNIQUE, btree (grandtot) WHERE grandtot < 0::numeric @@ -155,11 +155,11 @@ View definition: SET search_path = mvtest_mvschema, public; \d+ mvtest_tvm - Materialized view "mvtest_mvschema.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "mvtest_mvschema.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -356,11 +356,11 @@ UNION ALL CREATE MATERIALIZED VIEW mv_test2 AS SELECT moo, 2*moo FROM mvtest_vt2 UNION ALL SELECT moo, 3*moo FROM mvtest_vt2; \d+ mv_test2 - Materialized view "public.mv_test2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - moo | integer | | | | plain | | - ?column? | integer | | | | plain | | + Materialized view "public.mv_test2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + moo | integer | | | | plain | | | + ?column? | integer | | | | plain | | | View definition: SELECT mvtest_vt2.moo, 2 * mvtest_vt2.moo @@ -493,14 +493,14 @@ drop cascades to materialized view mvtest_mv_v_4 CREATE MATERIALIZED VIEW mv_unspecified_types AS SELECT 42 as i, 42.5 as num, 'foo' as u, 'foo'::unknown as u2, null as n; \d+ mv_unspecified_types - Materialized view "public.mv_unspecified_types" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - i | integer | | | | plain | | - num | numeric | | | | main | | - u | text | | | | extended | | - u2 | text | | | | extended | | - n | text | | | | extended | | + Materialized view "public.mv_unspecified_types" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + i | integer | | | | plain | | | + num | numeric | | | | main | pglz | | + u | text | | | | extended | pglz | | + u2 | text | | | | extended | pglz | | + n | text | | | | extended | pglz | | View definition: SELECT 42 AS i, 42.5 AS num, diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..d6912f2109 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -2813,34 +2813,34 @@ CREATE TABLE tbl_heap(f1 int, f2 char(100)) using heap; CREATE VIEW view_heap_psql AS SELECT f1 from tbl_heap_psql; CREATE MATERIALIZED VIEW mat_view_heap_psql USING heap_psql AS SELECT f1 from tbl_heap_psql; \d+ tbl_heap_psql - Table "tableam_display.tbl_heap_psql" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap_psql" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | \d+ tbl_heap - Table "tableam_display.tbl_heap" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | \set HIDE_TABLEAM off \d+ tbl_heap_psql - Table "tableam_display.tbl_heap_psql" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap_psql" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | Access method: heap_psql \d+ tbl_heap - Table "tableam_display.tbl_heap" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | Access method: heap -- AM is displayed for tables, indexes and materialized views. @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out index 63d6ab7a4e..71388c197d 100644 --- a/src/test/regress/expected/publication.out +++ b/src/test/regress/expected/publication.out @@ -74,11 +74,11 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall (1 row) \d+ testpub_tbl2 - Table "public.testpub_tbl2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl2_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl2_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl2_pkey" PRIMARY KEY, btree (id) Publications: @@ -187,22 +187,22 @@ ALTER PUBLICATION testpub_default SET TABLE testpub_tbl1; ALTER PUBLICATION testpub_default ADD TABLE pub_test.testpub_nopk; ALTER PUBLICATION testpib_ins_trunct ADD TABLE pub_test.testpub_nopk, testpub_tbl1; \d+ pub_test.testpub_nopk - Table "pub_test.testpub_nopk" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - foo | integer | | | | plain | | - bar | integer | | | | plain | | + Table "pub_test.testpub_nopk" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + foo | integer | | | | plain | | | + bar | integer | | | | plain | | | Publications: "testpib_ins_trunct" "testpub_default" "testpub_fortbl" \d+ testpub_tbl1 - Table "public.testpub_tbl1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl1_pkey" PRIMARY KEY, btree (id) Publications: @@ -224,11 +224,11 @@ ALTER PUBLICATION testpub_default DROP TABLE testpub_tbl1, pub_test.testpub_nopk ALTER PUBLICATION testpub_default DROP TABLE pub_test.testpub_nopk; ERROR: relation "testpub_nopk" is not part of the publication \d+ testpub_tbl1 - Table "public.testpub_tbl1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl1_pkey" PRIMARY KEY, btree (id) Publications: diff --git a/src/test/regress/expected/replica_identity.out b/src/test/regress/expected/replica_identity.out index 79002197a7..20ac0012ef 100644 --- a/src/test/regress/expected/replica_identity.out +++ b/src/test/regress/expected/replica_identity.out @@ -153,13 +153,13 @@ SELECT relreplident FROM pg_class WHERE oid = 'test_replica_identity'::regclass; (1 row) \d+ test_replica_identity - Table "public.test_replica_identity" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('test_replica_identity_id_seq'::regclass) | plain | | - keya | text | | not null | | extended | | - keyb | text | | not null | | extended | | - nonkey | text | | | | extended | | + Table "public.test_replica_identity" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('test_replica_identity_id_seq'::regclass) | plain | | | + keya | text | | not null | | extended | pglz | | + keyb | text | | not null | | extended | pglz | | + nonkey | text | | | | extended | pglz | | Indexes: "test_replica_identity_pkey" PRIMARY KEY, btree (id) "test_replica_identity_expr" UNIQUE, btree (keya, keyb, (3)) diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index 9506aaef82..7ebdc30ded 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -938,14 +938,14 @@ CREATE POLICY pp1 ON part_document AS PERMISSIVE CREATE POLICY pp1r ON part_document AS RESTRICTIVE TO regress_rls_dave USING (cid < 55); \d+ part_document - Partitioned table "regress_rls_schema.part_document" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ----------+---------+-----------+----------+---------+----------+--------------+------------- - did | integer | | | | plain | | - cid | integer | | | | plain | | - dlevel | integer | | not null | | plain | | - dauthor | name | | | | plain | | - dtitle | text | | | | extended | | + Partitioned table "regress_rls_schema.part_document" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +---------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + did | integer | | | | plain | | | + cid | integer | | | | plain | | | + dlevel | integer | | not null | | plain | | | + dauthor | name | | | | plain | | | + dtitle | text | | | | extended | pglz | | Partition key: RANGE (cid) Policies: POLICY "pp1" diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 6173473de9..8546f02c1c 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -3035,11 +3035,11 @@ select * from rules_log; create rule r3 as on delete to rules_src do notify rules_src_deletion; \d+ rules_src - Table "public.rules_src" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rules_src" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: r1 AS ON UPDATE TO rules_src DO INSERT INTO rules_log (f1, f2, tag) VALUES (old.f1,old.f2,'old'::text), (new.f1,new.f2,'new'::text) @@ -3055,11 +3055,11 @@ Rules: create rule r4 as on insert to rules_src do instead insert into rules_log AS trgt SELECT NEW.* RETURNING trgt.f1, trgt.f2; create rule r5 as on update to rules_src do instead UPDATE rules_log AS trgt SET tag = 'updated' WHERE trgt.f1 = new.f1; \d+ rules_src - Table "public.rules_src" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rules_src" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: r1 AS ON UPDATE TO rules_src DO INSERT INTO rules_log (f1, f2, tag) VALUES (old.f1,old.f2,'old'::text), (new.f1,new.f2,'new'::text) @@ -3086,11 +3086,11 @@ create rule rr as on update to rule_t1 do instead UPDATE rule_dest trgt SET (f2[1], f1, tag) = (SELECT new.f2, new.f1, 'updated'::varchar) WHERE trgt.f1 = new.f1 RETURNING new.*; \d+ rule_t1 - Table "public.rule_t1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rule_t1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: rr AS ON UPDATE TO rule_t1 DO INSTEAD UPDATE rule_dest trgt SET (f2[1], f1, tag) = ( SELECT new.f2, diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 431b3fa3de..a36b7e6069 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -125,11 +125,11 @@ SELECT stxname, stxdndistinct, stxddependencies, stxdmcv ALTER STATISTICS ab1_a_b_stats SET STATISTICS -1; \d+ ab1 - Table "public.ab1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | + Table "public.ab1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | Statistics objects: "public"."ab1_a_b_stats" (ndistinct, dependencies, mcv) ON a, b FROM ab1 diff --git a/src/test/regress/expected/update.out b/src/test/regress/expected/update.out index bf939d79f6..58e996fdee 100644 --- a/src/test/regress/expected/update.out +++ b/src/test/regress/expected/update.out @@ -711,14 +711,14 @@ DROP TRIGGER d15_insert_trig ON part_d_15_20; :init_range_parted; create table part_def partition of range_parted default; \d+ part_def - Table "public.part_def" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | bigint | | | | plain | | - c | numeric | | | | main | | - d | integer | | | | plain | | - e | character varying | | | | extended | | + Table "public.part_def" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | bigint | | | | plain | | | + c | numeric | | | | main | pglz | | + d | integer | | | | plain | | | + e | character varying | | | | extended | pglz | | Partition of: range_parted DEFAULT Partition constraint: (NOT ((a IS NOT NULL) AND (b IS NOT NULL) AND (((a = 'a'::text) AND (b >= '1'::bigint) AND (b < '10'::bigint)) OR ((a = 'a'::text) AND (b >= '10'::bigint) AND (b < '20'::bigint)) OR ((a = 'b'::text) AND (b >= '1'::bigint) AND (b < '10'::bigint)) OR ((a = 'b'::text) AND (b >= '10'::bigint) AND (b < '20'::bigint)) OR ((a = 'b'::text) AND (b >= '20'::bigint) AND (b < '30'::bigint))))) diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 94c5f023c6..458c297670 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -151,10 +151,10 @@ Indexes: Number of partitions: 2 (Use \d+ to list them.) \d+ testschema.part - Partitioned table "testschema.part" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Partitioned table "testschema.part" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition key: LIST (a) Indexes: "part_a_idx" btree (a), tablespace "regress_tblspace" @@ -171,10 +171,10 @@ Indexes: "part1_a_idx" btree (a), tablespace "regress_tblspace" \d+ testschema.part1 - Table "testschema.part1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Table "testschema.part1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition of: testschema.part FOR VALUES IN (1) Partition constraint: ((a IS NOT NULL) AND (a = 1)) Indexes: diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e0e1ef71dd..2407ca945b 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -116,6 +116,9 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +#test toast compression +test: compression + # event triggers cannot run concurrently with any test that runs DDL test: event_trigger # this test also uses event triggers, so likewise run it by itself diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 081fce32e7..152c8cb5b7 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -203,3 +203,4 @@ test: explain test: event_trigger test: fast_default test: stats +test: compression diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..0750a6c80e --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,74 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + + + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2f28de0355..9ebfe19a34 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 721b230bf2..49a62d6e33 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0002-Add-default_toast_compression-GUC.patch" From b49680d96644c4a83f081f490a354363f81d7091 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 29 Jan 2021 21:37:46 -0600 Subject: [PATCH v21 2/7] Add default_toast_compression GUC --- src/backend/access/common/tupdesc.c | 3 +- .../access/compression/compressamapi.c | 54 +++++++++++++++++++ src/backend/bootstrap/bootstrap.c | 7 ++- src/backend/commands/tablecmds.c | 4 +- src/backend/utils/cache/syscache.c | 6 ++- src/backend/utils/misc/guc.c | 12 +++++ src/include/access/compressamapi.h | 13 ++++- 7 files changed, 92 insertions(+), 7 deletions(-) diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index ca26fab487..b56b689493 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -25,6 +25,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" +#include "commands/defrem.h" #include "miscadmin.h" #include "parser/parse_type.h" #include "utils/acl.h" @@ -668,7 +669,7 @@ TupleDescInitEntry(TupleDesc desc, att->attcollation = typeForm->typcollation; if (IsStorageCompressible(typeForm->typstorage)) - att->attcompression = DefaultCompressionOid; + att->attcompression = get_compression_am_oid(default_toast_compression, false); else att->attcompression = InvalidOid; diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c index 663102c8d2..e8bfb23ec3 100644 --- a/src/backend/access/compression/compressamapi.c +++ b/src/backend/access/compression/compressamapi.c @@ -19,9 +19,12 @@ #include "access/htup_details.h" #include "access/reloptions.h" #include "access/table.h" +#include "commands/defrem.h" #include "utils/fmgroids.h" #include "utils/syscache.h" +char *default_toast_compression = DEFAULT_TOAST_COMPRESSION; // maybe needs pg_dump support ? + /* * CompressionOidToId - Convert compression Oid to built-in compression id. * @@ -59,3 +62,54 @@ CompressionIdToOid(CompressionId cmid) elog(ERROR, "Invalid compression method id %d", cmid); } } + +/* check_hook: validate new default_toast_compression */ +bool +check_default_toast_compression(char **newval, void **extra, GucSource source) +{ + if (**newval == '\0') + { + GUC_check_errdetail("%s cannot be empty.", + "default_toast_compression"); + return false; + } + + if (strlen(*newval) >= NAMEDATALEN) + { + GUC_check_errdetail("%s is too long (maximum %d characters).", + "default_toast_compression", NAMEDATALEN - 1); + return false; + } + + /* + * If we aren't inside a transaction, or not connected to a database, we + * cannot do the catalog access necessary to verify the method. Must + * accept the value on faith. + */ + if (IsTransactionState() && MyDatabaseId != InvalidOid) + { + if (!OidIsValid(get_compression_am_oid(*newval, true))) + { + /* + * When source == PGC_S_TEST, don't throw a hard error for a + * nonexistent table access method, only a NOTICE. See comments in + * guc.h. + */ + if (source == PGC_S_TEST) + { + ereport(NOTICE, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("compression access method \"%s\" does not exist", + *newval))); + } + else + { + GUC_check_errdetail("Compression access method \"%s\" does not exist.", + *newval); + return false; + } + } + } + + return true; +} diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 9b451eaa71..1b885f873a 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -29,6 +29,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/link-canary.h" +#include "commands/defrem.h" #include "libpq/pqsignal.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -732,8 +733,12 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) - attrtypes[attnum]->attcompression = DefaultCompressionOid; + { + /* Cannot call get_compression_am_oid this early */ + attrtypes[attnum]->attcompression = PGLZ_COMPRESSION_AM_OID; + } else attrtypes[attnum]->attcompression = InvalidOid; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index d7f4489c57..082db26bfa 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11925,7 +11925,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, if (!IsStorageCompressible(tform->typstorage)) attTup->attcompression = InvalidOid; else if (!OidIsValid(attTup->attcompression)) - attTup->attcompression = DefaultCompressionOid; + attTup->attcompression = get_compression_am_oid(default_toast_compression, false); } else attTup->attcompression = InvalidOid; @@ -17760,7 +17760,7 @@ GetAttributeCompression(Form_pg_attribute att, char *compression) /* fallback to default compression if it's not specified */ if (compression == NULL) - return DefaultCompressionOid; + return get_compression_am_oid(default_toast_compression, false); return get_compression_am_oid(compression, false); } diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c index e4dc4ee34e..74e42a3394 100644 --- a/src/backend/utils/cache/syscache.c +++ b/src/backend/utils/cache/syscache.c @@ -1127,8 +1127,10 @@ HeapTuple SearchSysCache1(int cacheId, Datum key1) { - Assert(cacheId >= 0 && cacheId < SysCacheSize && - PointerIsValid(SysCache[cacheId])); + Assert(cacheId >= 0); + Assert(cacheId < SysCacheSize); + // fprintf(stderr, "cache %d\n", cacheId); + Assert(PointerIsValid(SysCache[cacheId])); Assert(SysCache[cacheId]->cc_nkeys == 1); return SearchCatCache1(SysCache[cacheId], key1); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index eafdb1118e..70b26ea72a 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -30,6 +30,7 @@ #include #include "access/commit_ts.h" +#include "access/compressamapi.h" #include "access/gin.h" #include "access/rmgr.h" #include "access/tableam.h" @@ -3915,6 +3916,17 @@ static struct config_string ConfigureNamesString[] = check_default_table_access_method, NULL, NULL }, + { + {"default_toast_compression", PGC_USERSET, CLIENT_CONN_STATEMENT, + gettext_noop("Sets the default compression for new columns."), + NULL, + GUC_IS_NAME + }, + &default_toast_compression, + DEFAULT_TOAST_COMPRESSION, + check_default_toast_compression, NULL, NULL, NULL + }, + { {"default_tablespace", PGC_USERSET, CLIENT_CONN_STATEMENT, gettext_noop("Sets the default tablespace to create tables and indexes in."), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h index 8226ae0596..be5b4da7b8 100644 --- a/src/include/access/compressamapi.h +++ b/src/include/access/compressamapi.h @@ -15,8 +15,12 @@ #include "postgres.h" +#include "access/xact.h" #include "catalog/pg_am_d.h" +#include "miscadmin.h" #include "nodes/nodes.h" +#include "utils/guc.h" +#include "nodes/pg_list.h" /* * Built-in compression method-id. The toast compression header will store @@ -29,8 +33,11 @@ typedef enum CompressionId LZ4_COMPRESSION_ID = 1 } CompressionId; +#define DEFAULT_TOAST_COMPRESSION "pglz" + +extern char *default_toast_compression; + /* Use default compression method if it is not specified. */ -#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID #define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ (storage) != TYPSTORAGE_EXTERNAL) /* compression handler routines */ @@ -62,4 +69,8 @@ extern const CompressionAmRoutine lz4_compress_methods; extern CompressionId CompressionOidToId(Oid cmoid); extern Oid CompressionIdToOid(CompressionId cmid); +extern void assign_default_toast_compression(const char *newval, void **extra); +extern bool check_default_toast_compression(char **newval, void **extra, GucSource source); + + #endif /* COMPRESSAMAPI_H */ -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0003-alter-table-set-compression.patch" From 8e5276e8fd282078ca3e741b230fda897b9a6c7e Mon Sep 17 00:00:00 2001 From: dilipkumar Date: Fri, 18 Dec 2020 11:31:22 +0530 Subject: [PATCH v21 3/7] alter table set compression Add support for changing the compression method associated with a column, forcing a table rewrite. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov Discussions: https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com --- doc/src/sgml/ref/alter_table.sgml | 14 ++ src/backend/commands/tablecmds.c | 208 ++++++++++++++++---- src/backend/parser/gram.y | 9 + src/bin/psql/tab-complete.c | 2 +- src/include/commands/event_trigger.h | 1 + src/include/nodes/parsenodes.h | 3 +- src/test/regress/expected/compression.out | 59 ++++++ src/test/regress/expected/compression_1.out | 57 ++++++ src/test/regress/sql/compression.sql | 21 ++ 9 files changed, 329 insertions(+), 45 deletions(-) diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index c25ef5abd6..c4b53ec1c5 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -54,6 +54,7 @@ ALTER TABLE [ IF EXISTS ] name ALTER [ COLUMN ] column_name SET ( attribute_option = value [, ... ] ) ALTER [ COLUMN ] column_name RESET ( attribute_option [, ... ] ) ALTER [ COLUMN ] column_name SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN } + ALTER [ COLUMN ] column_name SET COMPRESSION compression_method ADD table_constraint [ NOT VALID ] ADD table_constraint_using_index ALTER CONSTRAINT constraint_name [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -383,6 +384,19 @@ WITH ( MODULUS numeric_literal, REM + + + SET COMPRESSION compression_method + + + + This clause adds compression to a column. The Compression method can be + set to any available compression method. The built-in methods are + pglz and lz4. + + + + ADD table_constraint [ NOT VALID ] diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 082db26bfa..89cca760c5 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -529,6 +529,8 @@ static void ATExecGenericOptions(Relation rel, List *options); static void ATExecEnableRowSecurity(Relation rel); static void ATExecDisableRowSecurity(Relation rel); static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls); +static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel, + const char *column, Node *newValue, LOCKMODE lockmode); static void index_copy_data(Relation rel, RelFileNode newrnode); static const char *storage_name(char c); @@ -3968,6 +3970,7 @@ AlterTableGetLockLevel(List *cmds) */ case AT_GenericOptions: case AT_AlterColumnGenericOptions: + case AT_SetCompression: cmd_lockmode = AccessExclusiveLock; break; @@ -4495,7 +4498,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, case AT_DisableRowSecurity: case AT_ForceRowSecurity: case AT_NoForceRowSecurity: - ATSimplePermissions(rel, ATT_TABLE); + case AT_SetCompression: + ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW); /* These commands never recurse */ /* No command-specific prep needed */ pass = AT_PASS_MISC; @@ -4903,6 +4907,10 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, Assert(rel->rd_rel->relkind == RELKIND_INDEX); ATExecAlterCollationRefreshVersion(rel, cmd->object); break; + case AT_SetCompression: + address = ATExecSetCompression(tab, rel, cmd->name, cmd->def, + lockmode); + break; default: /* oops */ elog(ERROR, "unrecognized alter table type: %d", (int) cmd->subtype); @@ -5527,6 +5535,9 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap, LOCKMODE lockmode) slot_getallattrs(oldslot); ExecClearTuple(newslot); + if (tab->rewrite & AT_REWRITE_ALTER_COMPRESSION) + CompareCompressionMethodAndDecompress(oldslot, newTupDesc); + /* copy attributes */ memcpy(newslot->tts_values, oldslot->tts_values, sizeof(Datum) * oldslot->tts_nvalid); @@ -7767,6 +7778,71 @@ ATExecSetOptions(Relation rel, const char *colName, Node *options, return address; } +/* + * Helper function for ATExecSetStorage and ATExecSetCompression + * + * Set the attcompression or attstorage for the respective index attribute. If + * attcompression is a valid oid then it will update the attcompression for the + * index attribute otherwise it will update the attstorage. + */ +static void +ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum, + Oid newcompression, char newstorage, LOCKMODE lockmode) +{ + HeapTuple tuple; + ListCell *lc; + Form_pg_attribute attrtuple; + + foreach(lc, RelationGetIndexList(rel)) + { + Oid indexoid = lfirst_oid(lc); + Relation indrel; + AttrNumber indattnum = 0; + + indrel = index_open(indexoid, lockmode); + + for (int i = 0; i < indrel->rd_index->indnatts; i++) + { + if (indrel->rd_index->indkey.values[i] == attnum) + { + indattnum = i + 1; + break; + } + } + + if (indattnum == 0) + { + index_close(indrel, lockmode); + continue; + } + + tuple = SearchSysCacheCopyAttNum(RelationGetRelid(indrel), indattnum); + + if (HeapTupleIsValid(tuple)) + { + attrtuple = (Form_pg_attribute) GETSTRUCT(tuple); + + /* + * If a valid compression method Oid is passed then update the + * attcompression, otherwise update the attstorage. + */ + if (OidIsValid(newcompression)) + attrtuple->attcompression = newcompression; + else + attrtuple->attstorage = newstorage; + + CatalogTupleUpdate(attrelation, &tuple->t_self, tuple); + + InvokeObjectPostAlterHook(RelationRelationId, + RelationGetRelid(rel), + attrtuple->attnum); + + heap_freetuple(tuple); + } + + index_close(indrel, lockmode); + } +} /* * ALTER TABLE ALTER COLUMN SET STORAGE * @@ -7782,7 +7858,6 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc Form_pg_attribute attrtuple; AttrNumber attnum; ObjectAddress address; - ListCell *lc; Assert(IsA(newValue, String)); storagemode = strVal(newValue); @@ -7846,47 +7921,8 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc * Apply the change to indexes as well (only for simple index columns, * matching behavior of index.c ConstructTupleDescriptor()). */ - foreach(lc, RelationGetIndexList(rel)) - { - Oid indexoid = lfirst_oid(lc); - Relation indrel; - AttrNumber indattnum = 0; - - indrel = index_open(indexoid, lockmode); - - for (int i = 0; i < indrel->rd_index->indnatts; i++) - { - if (indrel->rd_index->indkey.values[i] == attnum) - { - indattnum = i + 1; - break; - } - } - - if (indattnum == 0) - { - index_close(indrel, lockmode); - continue; - } - - tuple = SearchSysCacheCopyAttNum(RelationGetRelid(indrel), indattnum); - - if (HeapTupleIsValid(tuple)) - { - attrtuple = (Form_pg_attribute) GETSTRUCT(tuple); - attrtuple->attstorage = newstorage; - - CatalogTupleUpdate(attrelation, &tuple->t_self, tuple); - - InvokeObjectPostAlterHook(RelationRelationId, - RelationGetRelid(rel), - attrtuple->attnum); - - heap_freetuple(tuple); - } - - index_close(indrel, lockmode); - } + ApplyChangesToIndexes(rel, attrelation, attnum, InvalidOid, newstorage, + lockmode); table_close(attrelation, RowExclusiveLock); @@ -15033,6 +15069,92 @@ ATExecGenericOptions(Relation rel, List *options) heap_freetuple(tuple); } +/* + * ALTER TABLE ALTER COLUMN SET COMPRESSION + * + * Return value is the address of the modified column + */ +static ObjectAddress +ATExecSetCompression(AlteredTableInfo *tab, + Relation rel, + const char *column, + Node *newValue, + LOCKMODE lockmode) +{ + Relation attrel; + HeapTuple tuple; + Form_pg_attribute atttableform; + AttrNumber attnum; + char *compression; + char typstorage; + Oid cmoid; + Datum values[Natts_pg_attribute]; + bool nulls[Natts_pg_attribute]; + bool replace[Natts_pg_attribute]; + ObjectAddress address; + + Assert(IsA(newValue, String)); + compression = strVal(newValue); + + attrel = table_open(AttributeRelationId, RowExclusiveLock); + + tuple = SearchSysCacheAttName(RelationGetRelid(rel), column); + if (!HeapTupleIsValid(tuple)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("column \"%s\" of relation \"%s\" does not exist", + column, RelationGetRelationName(rel)))); + + /* Prevent them from altering a system attribute */ + atttableform = (Form_pg_attribute) GETSTRUCT(tuple); + attnum = atttableform->attnum; + if (attnum <= 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter system column \"%s\"", column))); + + typstorage = get_typstorage(atttableform->atttypid); + + /* Prevent from setting compression methods for uncompressible type */ + if (!IsStorageCompressible(typstorage)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(atttableform->atttypid)))); + + /* Initialize buffers for new tuple values */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replace, false, sizeof(replace)); + + /* Get the attribute compression method. */ + cmoid = GetAttributeCompression(atttableform, compression); + + if (atttableform->attcompression != cmoid) + tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION; + + atttableform->attcompression = cmoid; + CatalogTupleUpdate(attrel, &tuple->t_self, tuple); + + InvokeObjectPostAlterHook(RelationRelationId, + RelationGetRelid(rel), + atttableform->attnum); + + ReleaseSysCache(tuple); + + /* apply changes to the index column as well */ + ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', lockmode); + table_close(attrel, RowExclusiveLock); + + /* make changes visible */ + CommandCounterIncrement(); + + ObjectAddressSubSet(address, RelationRelationId, + RelationGetRelid(rel), atttableform->attnum); + return address; +} + + /* * Preparation phase for SET LOGGED/UNLOGGED * diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 355e273192..f2e9fe4ff8 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -2307,6 +2307,15 @@ alter_table_cmd: n->missing_ok = true; $$ = (Node *)n; } + /* ALTER TABLE ALTER [COLUMN] SET (COMPRESSION ) */ + | ALTER opt_column ColId SET optColumnCompression + { + AlterTableCmd *n = makeNode(AlterTableCmd); + n->subtype = AT_SetCompression; + n->name = $3; + n->def = (Node *) makeString($5);; + $$ = (Node *)n; + } /* ALTER TABLE DROP [COLUMN] IF EXISTS [RESTRICT|CASCADE] */ | DROP opt_column IF_P EXISTS ColId opt_drop_behavior { diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 17f7265038..7aeabf9543 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2098,7 +2098,7 @@ psql_completion(const char *text, int start, int end) /* ALTER TABLE ALTER [COLUMN] SET */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET")) - COMPLETE_WITH("(", "DEFAULT", "NOT NULL", "STATISTICS", "STORAGE"); + COMPLETE_WITH("(", "COMPRESSION", "DEFAULT", "NOT NULL", "STATISTICS", "STORAGE"); /* ALTER TABLE ALTER [COLUMN] SET ( */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "(")) diff --git a/src/include/commands/event_trigger.h b/src/include/commands/event_trigger.h index c11bf2d781..5a314f4c1d 100644 --- a/src/include/commands/event_trigger.h +++ b/src/include/commands/event_trigger.h @@ -32,6 +32,7 @@ typedef struct EventTriggerData #define AT_REWRITE_ALTER_PERSISTENCE 0x01 #define AT_REWRITE_DEFAULT_VAL 0x02 #define AT_REWRITE_COLUMN_REWRITE 0x04 +#define AT_REWRITE_ALTER_COMPRESSION 0x08 /* * EventTriggerData is the node type that is passed as fmgr "context" info diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 232c9f1248..ffb739f565 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -1875,7 +1875,8 @@ typedef enum AlterTableType AT_AddIdentity, /* ADD IDENTITY */ AT_SetIdentity, /* SET identity column options */ AT_DropIdentity, /* DROP IDENTITY */ - AT_AlterCollationRefreshVersion /* ALTER COLLATION ... REFRESH VERSION */ + AT_AlterCollationRefreshVersion, /* ALTER COLLATION ... REFRESH VERSION */ + AT_SetCompression /* SET COMPRESSION */ } AlterTableType; typedef struct ReplicaIdentityStmt diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 4dea269fc3..84d31c781b 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -142,6 +142,65 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); NOTICE: merging column "f1" with inherited definition ERROR: column "f1" has a compression method conflict DETAIL: pglz versus lz4 +-- test alter compression method with rewrite +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4; +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test alter compression method for the materialized view +ALTER TABLE cmdata1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4; +REFRESH MATERIALIZED VIEW mv; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | lz4 | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test alter compression method for the partioned table +ALTER TABLE cmpart ALTER COLUMN f1 SET COMPRESSION pglz; +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + pglz + lz4 +(2 rows) + -- check data is ok SELECT length(f1) FROM cmdata; length diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index a50c7679bb..3250c12859 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -139,6 +139,63 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); NOTICE: merging column "f1" with inherited definition ERROR: column "f1" has a compression method conflict DETAIL: pglz versus lz4 +-- test alter compression method with rewrite +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4; +ERROR: not built with lz4 support +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test alter compression method for the materialized view +ALTER TABLE cmdata1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4; +REFRESH MATERIALIZED VIEW mv; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | lz4 | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- +(0 rows) + +-- test alter compression method for the partioned table +ALTER TABLE cmpart ALTER COLUMN f1 SET COMPRESSION pglz; +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + pglz +(1 row) + +ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; +ERROR: not built with lz4 support +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + pglz +(1 row) + -- check data is ok SELECT length(f1) FROM cmdata; length diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 0750a6c80e..3b0da88c19 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -65,6 +65,27 @@ SELECT pg_column_compression(f1) FROM cmpart; CREATE TABLE cminh() INHERITS(cmdata, cmdata1); CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +-- test alter compression method with rewrite +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4; +\d+ cmdata +SELECT pg_column_compression(f1) FROM cmdata; + +-- test alter compression method for the materialized view +ALTER TABLE cmdata1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4; +REFRESH MATERIALIZED VIEW mv; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test alter compression method for the partioned table +ALTER TABLE cmpart ALTER COLUMN f1 SET COMPRESSION pglz; +SELECT pg_column_compression(f1) FROM cmpart; + +ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; +SELECT pg_column_compression(f1) FROM cmpart; + -- check data is ok SELECT length(f1) FROM cmdata; SELECT length(f1) FROM cmdata1; -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=iso-8859-1 Content-Disposition: attachment; filename="v21-0004-Add-support-for-PRESERVE.patch" Content-Transfer-Encoding: 8bit From d6026a83e26299a5f79a4151b78b150f7d4af13c Mon Sep 17 00:00:00 2001 From: dilipkumar Date: Mon, 4 Jan 2021 15:15:20 +0530 Subject: [PATCH v21 4/7] Add support for PRESERVE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Now the compression method can be changed without forcing a table rewrite, by including the old method in the PRESERVE list.  For supporting this the column will maintain the dependency with all the supported compression methods.  So whenever the compression method is altered the dependency is added with the new compression method and the dependency is removed for all the old compression methods which are not given in the preserve list.  If PRESERVE ALL is given then all the dependency is maintained. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Tomas Vondra and Robert Haas Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com --- doc/src/sgml/ref/alter_table.sgml | 10 +- src/backend/catalog/pg_depend.c | 7 + src/backend/commands/Makefile | 1 + src/backend/commands/compressioncmds.c | 293 ++++++++++++++++++++ src/backend/commands/tablecmds.c | 116 ++++---- src/backend/executor/nodeModifyTable.c | 13 +- src/backend/nodes/copyfuncs.c | 17 +- src/backend/nodes/equalfuncs.c | 15 +- src/backend/nodes/outfuncs.c | 15 +- src/backend/parser/gram.y | 52 +++- src/backend/parser/parse_utilcmd.c | 2 +- src/bin/pg_dump/pg_dump.c | 101 +++++++ src/bin/pg_dump/pg_dump.h | 14 +- src/bin/psql/tab-complete.c | 7 + src/include/commands/defrem.h | 7 + src/include/commands/tablecmds.h | 2 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 16 +- src/test/regress/expected/compression.out | 37 ++- src/test/regress/expected/compression_1.out | 39 ++- src/test/regress/expected/create_index.out | 56 ++-- src/test/regress/sql/compression.sql | 9 + 22 files changed, 732 insertions(+), 98 deletions(-) create mode 100644 src/backend/commands/compressioncmds.c diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index c4b53ec1c5..081fe078a4 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] name ALTER [ COLUMN ] column_name SET ( attribute_option = value [, ... ] ) ALTER [ COLUMN ] column_name RESET ( attribute_option [, ... ] ) ALTER [ COLUMN ] column_name SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN } - ALTER [ COLUMN ] column_name SET COMPRESSION compression_method + ALTER [ COLUMN ] column_name SET COMPRESSION compression_method [ PRESERVE (compression_preserve_list) | PRESERVE ALL ] ADD table_constraint [ NOT VALID ] ADD table_constraint_using_index ALTER CONSTRAINT constraint_name [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -386,13 +386,19 @@ WITH ( MODULUS numeric_literal, REM - SET COMPRESSION compression_method + SET COMPRESSION compression_method [ PRESERVE (compression_preserve_list) | PRESERVE ALL ] This clause adds compression to a column. The Compression method can be set to any available compression method. The built-in methods are pglz and lz4. + The PRESERVE list contains a list of compression + methods used in the column and determines which of them may be kept. + Without PRESERVE or if any of the pre-existing + compression methods are not preserved, the table will be rewritten. If + PRESERVE ALL is specified, then all of the existing + methods will be preserved and the table will not be rewritten. diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 63da24322d..dd376484b7 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -17,6 +17,7 @@ #include "access/genam.h" #include "access/htup_details.h" #include "access/table.h" +#include "catalog/pg_am.h" #include "catalog/dependency.h" #include "catalog/indexing.h" #include "catalog/pg_collation.h" @@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender, if (referenced->objectId == DEFAULT_COLLATION_OID) ignore_systempin = true; } + /* + * Record the dependency on compression access method for handling + * preserve. + */ + if (referenced->classId == AccessMethodRelationId) + ignore_systempin = true; } else Assert(!version); diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile index e8504f0ae4..a7395ad77d 100644 --- a/src/backend/commands/Makefile +++ b/src/backend/commands/Makefile @@ -21,6 +21,7 @@ OBJS = \ cluster.o \ collationcmds.o \ comment.o \ + compressioncmds.o \ constraint.o \ conversioncmds.o \ copy.o \ diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c new file mode 100644 index 0000000000..50f0f7faaa --- /dev/null +++ b/src/backend/commands/compressioncmds.c @@ -0,0 +1,293 @@ +/*------------------------------------------------------------------------- + * + * compressioncmds.c + * Routines for SQL commands for attribute compression methods + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/commands/compressioncmds.c + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/heapam.h" +#include "access/htup_details.h" +#include "access/reloptions.h" +#include "catalog/catalog.h" +#include "catalog/dependency.h" +#include "catalog/indexing.h" +#include "catalog/pg_attribute.h" +#include "catalog/pg_depend.h" +#include "commands/defrem.h" +#include "commands/tablecmds.h" +#include "miscadmin.h" +#include "nodes/parsenodes.h" +#include "utils/builtins.h" +#include "utils/fmgroids.h" +#include "utils/lsyscache.h" + +/* + * Get list of all supported compression methods for the given attribute. + * + * We maintain dependency of the attribute on the pg_am row for the current + * compression AM and all the preserved compression AM. So scan pg_depend and + * find the column dependency on the pg_am. Collect the list of access method + * oids on which this attribute has a dependency. + */ +static List * +lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids) +{ + LOCKMODE lock = AccessShareLock; + HeapTuple tup; + Relation rel; + SysScanDesc scan; + ScanKeyData key[3]; + List *cmoids = NIL; + + rel = table_open(DependRelationId, lock); + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(RelationRelationId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(attrelid)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum((int32) attnum)); + + scan = systable_beginscan(rel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup); + + if (depform->refclassid == AccessMethodRelationId) + cmoids = list_append_unique_oid(cmoids, depform->refobjid); + } + + systable_endscan(scan); + table_close(rel, lock); + + return cmoids; +} + +/* + * Remove the attribute dependency on the old compression methods + * + * Scan the pg_depend and search the attribute dependency on the pg_am. Remove + * dependency on previous am which is not preserved. The list of non-preserved + * AMs is given in cmoids. + */ +static void +remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids) +{ + LOCKMODE lock = AccessShareLock; + HeapTuple tup; + Relation rel; + SysScanDesc scan; + ScanKeyData key[3]; + + rel = table_open(DependRelationId, lock); + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(RelationRelationId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(attrelid)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum((int32) attnum)); + + scan = systable_beginscan(rel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup); + + if (list_member_oid(cmoids, depform->refobjid)) + { + Assert(depform->refclassid == AccessMethodRelationId); + CatalogTupleDelete(rel, &tup->t_self); + } + } + + systable_endscan(scan); + table_close(rel, lock); +} + +/* + * Check whether the given compression method oid is supported by + * the target attribute. + */ +bool +IsCompressionSupported(Form_pg_attribute att, Oid cmoid) +{ + List *cmoids = NIL; + + /* Check whether it is same as the current compression oid */ + if (cmoid == att->attcompression) + return true; + + /* Check the oid in all preserved compresion methods */ + cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL); + if (list_member_oid(cmoids, cmoid)) + return true; + else + return false; +} + +/* + * In binary upgrade mode add the dependencies for all the preserved compression + * method. + */ +static void +BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve) +{ + ListCell *cell; + + foreach(cell, preserve) + { + char *cmname_p = strVal(lfirst(cell)); + Oid cmoid_p = get_compression_am_oid(cmname_p, false); + + add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p); + } +} + +/* + * Get the compression method oid based on the compression method name. When + * compression is not specified returns default attribute compression. It is + * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION + * syntax is optional. + * + * For ALTER command, check all the supported compression methods for the + * attribute and if the preserve list is not passed or some of the old + * compression methods are not given in the preserved list then delete + * dependency from the old compression methods and force the table rewrite. + */ +Oid +GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression, + bool *need_rewrite) +{ + Oid cmoid; + char typstorage = get_typstorage(att->atttypid); + ListCell *cell; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + return InvalidOid; + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return get_compression_am_oid(default_toast_compression, false); + + cmoid = get_compression_am_oid(compression->cmname, false); + + /* + * Determine if the column needs rewrite or not. Rewrite conditions: SET + * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not + * with full list of previous access methods. + */ + if (need_rewrite != NULL) + { + List *previous_cmoids = NIL; + + *need_rewrite = false; + + /* + * In binary upgrade mode, just create a dependency on all preserved + * methods. + */ + if (IsBinaryUpgrade) + { + BinaryUpgradeAddPreserve(att, compression->preserve); + return cmoid; + } + + /* If we have preserved all then rewrite is not required */ + if (compression->preserve_all) + return cmoid; + + previous_cmoids = + lookup_attribute_compression(att->attrelid, att->attnum, NULL); + + foreach(cell, compression->preserve) + { + char *cmname_p = strVal(lfirst(cell)); + Oid cmoid_p = get_compression_am_oid(cmname_p, false); + + if (!list_member_oid(previous_cmoids, cmoid_p)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("\"%s\" compression access method cannot be preserved", cmname_p), + errhint("use \"pg_column_compression\" function for list of compression methods"))); + + /* + * Remove from previous list, also protect from duplicate + * entries in the PRESERVE list + */ + previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p); + } + + /* delete the current cmoid from the list */ + previous_cmoids = list_delete_oid(previous_cmoids, cmoid); + + /* + * If the list of previous Oids is not empty after deletions then + * we need to rewrite tuples in the table. Also remove the dependency + * on the old compression methods which are no longer preserved. + */ + if (list_length(previous_cmoids) != 0) + { + remove_old_dependencies(att->attrelid, att->attnum, + previous_cmoids); + *need_rewrite = true; + } + + /* Cleanup */ + list_free(previous_cmoids); + } + + return cmoid; +} + +/* + * Construct ColumnCompression node from the compression method oid. + */ +ColumnCompression * +MakeColumnCompression(Oid attcompression) +{ + ColumnCompression *node; + + if (!OidIsValid(attcompression)) + return NULL; + + node = makeNode(ColumnCompression); + node->cmname = get_am_name(attcompression); + + return node; +} diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 89cca760c5..9d732fc167 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel); static void ATExecDisableRowSecurity(Relation rel); static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls); static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel, - const char *column, Node *newValue, LOCKMODE lockmode); + const char *column, + ColumnCompression *compression, + LOCKMODE lockmode); static void index_copy_data(Relation rel, RelFileNode newrnode); static const char *storage_name(char c); @@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); -static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -587,6 +588,7 @@ ObjectAddress DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, ObjectAddress *typaddress, const char *queryString) { + int i; char relname[NAMEDATALEN]; Oid namespaceId; Oid relationId; @@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, relkind == RELKIND_PARTITIONED_TABLE || relkind == RELKIND_MATVIEW) attr->attcompression = - GetAttributeCompression(attr, colDef->compression); + GetAttributeCompression(attr, colDef->compression, NULL); else attr->attcompression = InvalidOid; } @@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, */ rel = relation_open(relationId, AccessExclusiveLock); + /* + * Add the dependency on the respective compression AM for the relation + * attributes. + */ + for (i = 0; i < (RelationGetDescr(rel))->natts; i++) + { + Form_pg_attribute attr; + + attr = TupleDescAttr(RelationGetDescr(rel), i); + if (OidIsValid(attr->attcompression)) + add_column_compression_dependency(attr->attrelid, attr->attnum, + attr->attcompression); + } + /* * Now add any newly specified column default and generation expressions * to the new relation. These are passed to us in the form of raw @@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence, /* Copy/check compression parameter */ if (OidIsValid(attribute->attcompression)) { - char *compression = get_am_name(attribute->attcompression); + ColumnCompression *compression = + MakeColumnCompression(attribute->attcompression); if (!def->compression) def->compression = compression; - else if (strcmp(def->compression, compression)) + else if (strcmp(def->compression->cmname, compression->cmname)) ereport(ERROR, (errcode(ERRCODE_DATATYPE_MISMATCH), errmsg("column \"%s\" has a compression method conflict", attributeName), - errdetail("%s versus %s", def->compression, compression))); + errdetail("%s versus %s", def->compression->cmname, compression->cmname))); } def->inhcount++; @@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; - def->compression = get_am_name(attribute->attcompression); + def->compression = MakeColumnCompression( + attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->compression = newdef->compression; else if (newdef->compression) { - if (strcmp(def->compression, newdef->compression)) + if (strcmp(def->compression->cmname, newdef->compression->cmname)) ereport(ERROR, (errcode(ERRCODE_DATATYPE_MISMATCH), errmsg("column \"%s\" has a compression method conflict", attributeName), - errdetail("%s versus %s", def->compression, newdef->compression))); + errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname))); } /* Mark the column as locally defined */ @@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, ATExecAlterCollationRefreshVersion(rel, cmd->object); break; case AT_SetCompression: - address = ATExecSetCompression(tab, rel, cmd->name, cmd->def, + address = ATExecSetCompression(tab, rel, cmd->name, + (ColumnCompression *) cmd->def, lockmode); break; default: /* oops */ @@ -6402,7 +6421,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, if (rel->rd_rel->relkind == RELKIND_RELATION || rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) attribute.attcompression = GetAttributeCompression(&attribute, - colDef->compression); + colDef->compression, + NULL); else attribute.attcompression = InvalidOid; @@ -6577,6 +6597,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, */ add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid); add_column_collation_dependency(myrelid, newattnum, attribute.attcollation); + add_column_compression_dependency(myrelid, newattnum, attribute.attcompression); /* * Propagate to children as appropriate. Unlike most other ALTER @@ -6724,6 +6745,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid) } } +/* + * Install a dependency for compression on its column. + * + * This is used for identifying all the supported compression methods + * (current and preserved) for a attribute. + * + * If dependency is already there the whole thing is skipped. + */ +void +add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid) +{ + ObjectAddress acref, + attref; + + Assert(relid > 0 && attnum > 0); + + ObjectAddressSet(acref, AccessMethodRelationId, cmoid); + ObjectAddressSubSet(attref, RelationRelationId, relid, attnum); + + recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true); +} + /* * ALTER TABLE ALTER COLUMN DROP NOT NULL */ @@ -11859,7 +11902,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, foundDep->refobjid == attTup->attcollation) && !(foundDep->refclassid == RelationRelationId && foundDep->refobjid == RelationGetRelid(rel) && - foundDep->refobjsubid != 0) + foundDep->refobjsubid != 0) && + foundDep->refclassid != AccessMethodRelationId ) elog(ERROR, "found unexpected dependency for column: %s", getObjectDescription(&foundObject, false)); @@ -11974,6 +12018,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype); add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid); + /* Create dependency for new attribute compression */ + if (OidIsValid(attTup->attcompression)) + add_column_compression_dependency(RelationGetRelid(rel), attnum, + attTup->attcompression); + /* * Drop any pg_statistic entry for the column, since it's now wrong type */ @@ -15078,24 +15127,21 @@ static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel, const char *column, - Node *newValue, + ColumnCompression *compression, LOCKMODE lockmode) { Relation attrel; HeapTuple tuple; Form_pg_attribute atttableform; AttrNumber attnum; - char *compression; char typstorage; Oid cmoid; + bool need_rewrite; Datum values[Natts_pg_attribute]; bool nulls[Natts_pg_attribute]; bool replace[Natts_pg_attribute]; ObjectAddress address; - Assert(IsA(newValue, String)); - compression = strVal(newValue); - attrel = table_open(AttributeRelationId, RowExclusiveLock); tuple = SearchSysCacheAttName(RelationGetRelid(rel), column); @@ -15128,11 +15174,16 @@ ATExecSetCompression(AlteredTableInfo *tab, memset(replace, false, sizeof(replace)); /* Get the attribute compression method. */ - cmoid = GetAttributeCompression(atttableform, compression); + cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite); if (atttableform->attcompression != cmoid) + add_column_compression_dependency(atttableform->attrelid, + atttableform->attnum, cmoid); + if (need_rewrite) tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION; + atttableform->attcompression = cmoid; + atttableform->attcompression = cmoid; CatalogTupleUpdate(attrel, &tuple->t_self, tuple); @@ -17857,32 +17908,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } - -/* - * Get compression method Oid for the attribute. - */ -static Oid -GetAttributeCompression(Form_pg_attribute att, char *compression) -{ - char typstorage = get_typstorage(att->atttypid); - - /* - * No compression for the plain/external storage, refer comments atop - * attcompression parameter in pg_attribute.h - */ - if (!IsStorageCompressible(typstorage)) - { - if (compression != NULL) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column data type %s does not support compression", - format_type_be(att->atttypid)))); - return InvalidOid; - } - - /* fallback to default compression if it's not specified */ - if (compression == NULL) - return get_compression_am_oid(default_toast_compression, false); - - return get_compression_am_oid(compression, false); -} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 684e836071..916f6541c1 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -45,6 +45,7 @@ #include "access/toast_internals.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/defrem.h" #include "commands/trigger.h" #include "executor/execPartition.h" #include "executor/executor.h" @@ -2054,6 +2055,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot, int natts = slot->tts_tupleDescriptor->natts; bool isnull = false; bool decompressed_any = false; + Oid cmoid = InvalidOid; TupleDesc tupleDesc = slot->tts_tupleDescriptor; if (natts == 0) @@ -2062,7 +2064,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot, /* * Loop for all the attributes in the tuple and check if any of the * attribute is compressed in the source tuple and its compression method - * is not same as the target compression method then we need to decompress + * is not supported by the target attribute then we need to decompress * it. */ for (i = 0; i < natts; i++) @@ -2083,11 +2085,12 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot, continue; /* - * Get the compression method stored in the toast header and - * compare with the compression method of the target. + * Get the compression method stored in the toast header and if the + * compression method is not supported by the target attribute then + * we need to decompress it. */ - if (targetTupDesc->attrs[i].attcompression != - CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value))) + cmoid = CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value)); + if (!IsCompressionSupported(&targetTupDesc->attrs[i], cmoid)) { new_value = detoast_attr(new_value); slot->tts_values[attnum - 1] = PointerGetDatum(new_value); diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 3ea32323cd..6588c84c4c 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2932,7 +2932,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); - COPY_STRING_FIELD(compression); + COPY_NODE_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); @@ -2952,6 +2952,18 @@ _copyColumnDef(const ColumnDef *from) return newnode; } +static ColumnCompression * +_copyColumnCompression(const ColumnCompression *from) +{ + ColumnCompression *newnode = makeNode(ColumnCompression); + + COPY_STRING_FIELD(cmname); + COPY_SCALAR_FIELD(preserve_all); + COPY_NODE_FIELD(preserve); + + return newnode; +} + static Constraint * _copyConstraint(const Constraint *from) { @@ -5640,6 +5652,9 @@ copyObjectImpl(const void *from) case T_ColumnDef: retval = _copyColumnDef(from); break; + case T_ColumnCompression: + retval = _copyColumnCompression(from); + break; case T_Constraint: retval = _copyConstraint(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 227bb07bbd..3aa6176a30 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2598,7 +2598,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); - COMPARE_STRING_FIELD(compression); + COMPARE_NODE_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); @@ -2618,6 +2618,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) return true; } +static bool +_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b) +{ + COMPARE_STRING_FIELD(cmname); + COMPARE_SCALAR_FIELD(preserve_all); + COMPARE_NODE_FIELD(preserve); + + return true; +} + static bool _equalConstraint(const Constraint *a, const Constraint *b) { @@ -3693,6 +3703,9 @@ equal(const void *a, const void *b) case T_ColumnDef: retval = _equalColumnDef(a, b); break; + case T_ColumnCompression: + retval = _equalColumnCompression(a, b); + break; case T_Constraint: retval = _equalConstraint(a, b); break; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 187d3570e5..4f7053d63b 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2865,7 +2865,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); - WRITE_STRING_FIELD(compression); + WRITE_NODE_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); @@ -2883,6 +2883,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_LOCATION_FIELD(location); } +static void +_outColumnCompression(StringInfo str, const ColumnCompression *node) +{ + WRITE_NODE_TYPE("COLUMNCOMPRESSION"); + + WRITE_STRING_FIELD(cmname); + WRITE_BOOL_FIELD(preserve_all); + WRITE_NODE_FIELD(preserve); +} + static void _outTypeName(StringInfo str, const TypeName *node) { @@ -4230,6 +4240,9 @@ outNode(StringInfo str, const void *obj) case T_ColumnDef: _outColumnDef(str, obj); break; + case T_ColumnCompression: + _outColumnCompression(str, obj); + break; case T_TypeName: _outTypeName(str, obj); break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index f2e9fe4ff8..332e60f170 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -595,7 +595,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type hash_partbound %type hash_partbound_elem -%type optColumnCompression +%type optColumnCompression alterColumnCompression +%type compressionClause +%type optCompressionPreserve /* * Non-keyword token types. These are hard-wired into the "flex" lexer. @@ -2308,12 +2310,12 @@ alter_table_cmd: $$ = (Node *)n; } /* ALTER TABLE ALTER [COLUMN] SET (COMPRESSION ) */ - | ALTER opt_column ColId SET optColumnCompression + | ALTER opt_column ColId SET alterColumnCompression { AlterTableCmd *n = makeNode(AlterTableCmd); n->subtype = AT_SetCompression; n->name = $3; - n->def = (Node *) makeString($5);; + n->def = $5; $$ = (Node *)n; } /* ALTER TABLE DROP [COLUMN] IF EXISTS [RESTRICT|CASCADE] */ @@ -3436,7 +3438,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; - n->compression = $3; + n->compression = (ColumnCompression *) $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3491,13 +3493,43 @@ columnOptions: ColId ColQualList } ; +optCompressionPreserve: + PRESERVE '(' name_list ')' { $$ = $3; } + | /*EMPTY*/ { $$ = NULL; } + ; + +compressionClause: + COMPRESSION name { $$ = pstrdup($2); } + ; + optColumnCompression: - COMPRESSION name - { - $$ = $2; - } - | /*EMPTY*/ { $$ = NULL; } - ; + compressionClause + { + ColumnCompression *n = makeNode(ColumnCompression); + n->cmname = $1; + n->preserve = NIL; + $$ = (Node *) n; + } + | /*EMPTY*/ { $$ = NULL; } + ; + +alterColumnCompression: + compressionClause optCompressionPreserve + { + ColumnCompression *n = makeNode(ColumnCompression); + n->cmname = $1; + n->preserve = (List *) $2; + $$ = (Node *) n; + } + | compressionClause PRESERVE ALL + { + ColumnCompression *n = makeNode(ColumnCompression); + n->cmname = $1; + n->preserve_all = true; + n->preserve = NIL; + $$ = (Node *) n; + } + ; ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index e2ac159aa2..a2a6b218f3 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla /* Likewise, copy compression if requested */ if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION && OidIsValid(attribute->attcompression)) - def->compression = get_am_name(attribute->attcompression); + def->compression = MakeColumnCompression(attribute->attcompression); else def->compression = NULL; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index becf565cdd..c09f3972c6 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -9010,6 +9010,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) } PQclear(res); } + + /* + * Get compression info + */ + if (fout->remoteVersion >= 140000 && dopt->binary_upgrade) + { + int i_amname; + int i_amoid; + int i_curattnum; + int start; + + pg_log_info("finding compression info for table \"%s.%s\"", + tbinfo->dobj.namespace->dobj.name, + tbinfo->dobj.name); + + tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *)); + + resetPQExpBuffer(q); + appendPQExpBuffer(q, + " SELECT attrelid::pg_catalog.regclass AS relname, attname," + " amname, am.oid as amoid, d.objsubid AS curattnum" + " FROM pg_depend d" + " JOIN pg_attribute a ON" + " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid" + " AND a.attnum = d.objsubid AND d.deptype = 'n'" + " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)" + " JOIN pg_am am ON" + " (d.deptype = 'n' AND d.refobjid = am.oid)" + " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);", + tbinfo->dobj.catId.oid); + + res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK); + ntups = PQntuples(res); + + if (ntups > 0) + { + int j; + int k; + + i_amname = PQfnumber(res, "amname"); + i_amoid = PQfnumber(res, "amoid"); + i_curattnum = PQfnumber(res, "curattnum"); + + start = 0; + + for (j = 0; j < ntups; j++) + { + int attnum = atoi(PQgetvalue(res, j, i_curattnum)); + + if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum) + { + AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo)); + + cminfo->nitems = j - start + 1; + cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems); + + for (k = start; k < start + cminfo->nitems; k++) + { + AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem)); + + cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname)); + cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid)); + + cminfo->items[k - start] = cmitem; + } + + tbinfo->attcompression[attnum - 1] = cminfo; + start = j + 1; /* start from next */ + } + } + } + + PQclear(res); + } } destroyPQExpBuffer(q); @@ -16326,6 +16400,33 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attfdwoptions[j]); + + /* + * Dump per-column compression options + */ + if (tbinfo->attcompression && tbinfo->attcompression[j]) + { + AttrCompressionInfo *cminfo = tbinfo->attcompression[j]; + + appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s", + qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]); + + if (cminfo->nitems > 0) + { + appendPQExpBuffer(q, "\nPRESERVE ("); + for (int i = 0; i < cminfo->nitems; i++) + { + AttrCompressionItem *item = cminfo->items[i]; + + if (i == 0) + appendPQExpBuffer(q, "%s", item->amname); + else + appendPQExpBuffer(q, ", %s", item->amname); + } + appendPQExpBuffer(q, ")"); + } + appendPQExpBuffer(q, ";\n"); + } } if (ftoptions) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 53ece7abd7..2a751321f5 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -328,7 +328,7 @@ typedef struct _tableInfo char *amname; /* relation access method */ char **attcmnames; /* per-attribute current compression method */ - + struct _attrCompressionInfo **attcompression; /* per-attribute all compression data */ /* * Stuff computed only for dumpable tables. */ @@ -357,6 +357,18 @@ typedef struct _attrDefInfo bool separate; /* true if must dump as separate item */ } AttrDefInfo; +typedef struct _attrCompressionItem +{ + Oid amoid; /* attribute compression oid */ + char *amname; /* compression access method name */ +} AttrCompressionItem; + +typedef struct _attrCompressionInfo +{ + int nitems; + AttrCompressionItem **items; +} AttrCompressionInfo; + typedef struct _tableDataInfo { DumpableObject dobj; diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 7aeabf9543..c40d498a30 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end) else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "(")) COMPLETE_WITH("n_distinct", "n_distinct_inherited"); + /* ALTER TABLE ALTER [COLUMN] SET COMPRESSION */ + else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) || + Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny)) + COMPLETE_WITH("PRESERVE"); + else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") || + Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE")) + COMPLETE_WITH("( ", "ALL"); /* ALTER TABLE ALTER [COLUMN] SET STORAGE */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE")) diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index e5aea8a240..bd53f9bb0f 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); +/* commands/compressioncmds.c */ +extern Oid GetAttributeCompression(Form_pg_attribute att, + ColumnCompression *compression, + bool *need_rewrite); +extern ColumnCompression *MakeColumnCompression(Oid atttcompression); +extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid); + /* support routines in commands/define.c */ extern char *defGetString(DefElem *def); diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h index b3d30acc35..e6c98e65d4 100644 --- a/src/include/commands/tablecmds.h +++ b/src/include/commands/tablecmds.h @@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); extern bool PartConstraintImpliedByRelConstraint(Relation scanrel, List *partConstraint); +extern void add_column_compression_dependency(Oid relid, int32 attnum, + Oid cmoid); #endif /* TABLECMDS_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index ded2ad9692..d630c54efa 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -479,6 +479,7 @@ typedef enum NodeTag T_PartitionBoundSpec, T_PartitionRangeDatum, T_PartitionCmd, + T_ColumnCompression, T_VacuumRelation, /* diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ffb739f565..fd9c6abddf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -623,6 +623,20 @@ typedef struct RangeTableSample int location; /* method name location, or -1 if unknown */ } RangeTableSample; +/* + * ColumnCompression - compression parameters for some attribute + * + * This represents compression information defined using clause: + * .. COMPRESSION PRESERVE + */ +typedef struct ColumnCompression +{ + NodeTag type; + char *cmname; + bool preserve_all; + List *preserve; +} ColumnCompression; + /* * ColumnDef - column definition (used in various creates) * @@ -646,7 +660,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ - char *compression; /* compression method for column */ + ColumnCompression *compression; /* column compression */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 84d31c781b..151478bcdd 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -201,12 +201,47 @@ SELECT pg_column_compression(f1) FROM cmpart; lz4 (2 rows) +-- preserve old compression method +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4); +INSERT INTO cmdata VALUES (repeat('1234567890',1004)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL; +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + -- check data is ok SELECT length(f1) FROM cmdata; length -------- 10000 -(1 row) + 10040 +(2 rows) SELECT length(f1) FROM cmdata1; length diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index 3250c12859..81182fe949 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -196,12 +196,49 @@ SELECT pg_column_compression(f1) FROM cmpart; pglz (1 row) +-- preserve old compression method +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4); +ERROR: "lz4" compression access method cannot be preserved +HINT: use "pg_column_compression" function for list of compression methods +INSERT INTO cmdata VALUES (repeat('1234567890',1004)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL; +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + -- check data is ok SELECT length(f1) FROM cmdata; length -------- 10000 -(1 row) + 10040 +(2 rows) SELECT length(f1) FROM cmdata1; length diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ce734f7ef3..b61c6278a2 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND 'concur_reindex_ind4'::regclass, 'concur_reindex_matview'::regclass) ORDER BY 1, 2; - obj | objref | deptype -------------------------------------------+------------------------------------------------------------+--------- - index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i - index concur_reindex_ind2 | collation "default" | n - index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a - index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a - index concur_reindex_ind3 | table concur_reindex_tab | a - index concur_reindex_ind4 | collation "default" | n - index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a - index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a - materialized view concur_reindex_matview | schema public | n - table concur_reindex_tab | schema public | n -(10 rows) + obj | objref | deptype +-------------------------------------------------------+------------------------------------------------------------+--------- + column c2 of materialized view concur_reindex_matview | access method pglz | n + column c2 of table concur_reindex_tab | access method pglz | n + index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n + index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a + index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a + index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n + index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a + index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a + materialized view concur_reindex_matview | schema public | n + table concur_reindex_tab | schema public | n +(12 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND 'concur_reindex_ind4'::regclass, 'concur_reindex_matview'::regclass) ORDER BY 1, 2; - obj | objref | deptype -------------------------------------------+------------------------------------------------------------+--------- - index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i - index concur_reindex_ind2 | collation "default" | n - index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a - index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a - index concur_reindex_ind3 | table concur_reindex_tab | a - index concur_reindex_ind4 | collation "default" | n - index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a - index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a - materialized view concur_reindex_matview | schema public | n - table concur_reindex_tab | schema public | n -(10 rows) + obj | objref | deptype +-------------------------------------------------------+------------------------------------------------------------+--------- + column c2 of materialized view concur_reindex_matview | access method pglz | n + column c2 of table concur_reindex_tab | access method pglz | n + index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n + index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a + index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a + index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n + index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a + index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a + materialized view concur_reindex_matview | schema public | n + table concur_reindex_tab | schema public | n +(12 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 3b0da88c19..e447316573 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -86,6 +86,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; SELECT pg_column_compression(f1) FROM cmpart; +-- preserve old compression method +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4); +INSERT INTO cmdata VALUES (repeat('1234567890',1004)); +\d+ cmdata +SELECT pg_column_compression(f1) FROM cmdata; +\d+ cmdata +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL; +SELECT pg_column_compression(f1) FROM cmdata; + -- check data is ok SELECT length(f1) FROM cmdata; SELECT length(f1) FROM cmdata1; -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0005-Create-custom-compression-methods.patch" From d00579f5a4bf5ea6c568f96998cd17a7313fd838 Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Fri, 29 Jan 2021 15:44:09 +0530 Subject: [PATCH v21 5/7] Create custom compression methods Provide syntax to create custom compression methods. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Tomas Vondra and Robert Haas Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby --- doc/src/sgml/ref/alter_table.sgml | 7 ++- doc/src/sgml/ref/create_access_method.sgml | 13 ++-- doc/src/sgml/ref/create_table.sgml | 3 +- src/backend/access/common/detoast.c | 59 +++++++++++++++++-- src/backend/access/common/toast_internals.c | 19 +++++- src/backend/access/compression/compress_lz4.c | 21 +++---- .../access/compression/compress_pglz.c | 20 +++---- .../access/compression/compressamapi.c | 27 +++++++++ src/backend/access/index/amapi.c | 50 ++++++++++++---- src/backend/commands/amcmds.c | 5 ++ src/backend/executor/nodeModifyTable.c | 2 +- src/backend/parser/gram.y | 1 + src/backend/utils/adt/varlena.c | 7 +-- src/bin/pg_dump/pg_dump.c | 3 + src/bin/psql/tab-complete.c | 6 ++ src/include/access/amapi.h | 1 + src/include/access/compressamapi.h | 16 +++-- src/include/access/detoast.h | 8 +++ src/include/access/toast_internals.h | 15 +++++ src/test/regress/expected/compression.out | 40 ++++++++++++- src/test/regress/expected/compression_1.out | 41 ++++++++++++- src/test/regress/sql/compression.sql | 10 ++++ 22 files changed, 312 insertions(+), 62 deletions(-) diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index 081fe078a4..46f34bffe5 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -390,9 +390,10 @@ WITH ( MODULUS numeric_literal, REM - This clause adds compression to a column. The Compression method can be - set to any available compression method. The built-in methods are - pglz and lz4. + This clause adds compression to a column. The Compression method could be + created with or + it can be set to any available compression method. The built-in methods + are pglz and lz4. The PRESERVE list contains a list of compression methods used in the column and determines which of them may be kept. Without PRESERVE or if any of the pre-existing diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml index dae43dbaed..79f1290a58 100644 --- a/doc/src/sgml/ref/create_access_method.sgml +++ b/doc/src/sgml/ref/create_access_method.sgml @@ -61,7 +61,7 @@ CREATE ACCESS METHOD name This clause specifies the type of access method to define. - Only TABLE and INDEX + TABLE, INDEX and COMPRESSION are supported at present. @@ -77,12 +77,15 @@ CREATE ACCESS METHOD name declared to take a single argument of type internal, and its return type depends on the type of access method; for TABLE access methods, it must - be table_am_handler and for INDEX - access methods, it must be index_am_handler. + be table_am_handler, for INDEX + access methods, it must be index_am_handler and + for COMPRESSION access methods, it must be + compression_am_handler. The C-level API that the handler function must implement varies depending on the type of access method. The table access method API - is described in and the index access method - API is described in . + is described in , the index access method + API is described in an the compression access + method API is described in . diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index d514c7f8c1..10b468fb2d 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -998,7 +998,8 @@ WITH ( MODULUS numeric_literal, REM This clause adds the compression method to a column. The Compression - method can be set from available compression methods. The built-in + method could be created with or + it can be set from the available compression methods. The built-in methods are pglz and lz4. If no compression method is specified, then compressible types will have the default compression method pglz. diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index c90464053c..e032c30d25 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -457,13 +457,44 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_oid - + * + * Return the Oid of the compresion method stored in the compressed data. + * For built-in methods, we only store the built-in compression method id in + * first 2-bits of the rawsize and that is directly mapped to the compression + * method Oid. And, for the custom compression method we store the Oid of the + * compression method in the custom compression header. + */ +Oid +toast_get_compression_oid(struct varlena *attr) +{ + CompressionId cmid; + + /* + * If it is custom compression id then get the Oid from the custom + * compression header otherwise, directly translate the built-in + * compression id to compression method Oid. + */ + cmid = TOAST_COMPRESS_METHOD(attr); + if (IsCustomCompression(cmid)) + { + toast_compress_header_custom *hdr; + + hdr = (toast_compress_header_custom *) attr; + return hdr->cmoid; + } + else + return CompressionIdToOid(cmid); +} + /* ---------- * toast_get_compression_handler - get the compression handler routines * * helper function for toast_decompress_datum and toast_decompress_datum_slice */ static inline const CompressionAmRoutine * -toast_get_compression_handler(struct varlena *attr) +toast_get_compression_handler(struct varlena *attr, int32 *header_size) { const CompressionAmRoutine *cmroutine; CompressionId cmid; @@ -477,10 +508,21 @@ toast_get_compression_handler(struct varlena *attr) { case PGLZ_COMPRESSION_ID: cmroutine = &pglz_compress_methods; + *header_size = TOAST_COMPRESS_HDRSZ; break; case LZ4_COMPRESSION_ID: cmroutine = &lz4_compress_methods; + *header_size = TOAST_COMPRESS_HDRSZ; break; + case CUSTOM_COMPRESSION_ID: + { + toast_compress_header_custom *hdr; + + hdr = (toast_compress_header_custom *) attr; + cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid); + *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ; + break; + } default: elog(ERROR, "Invalid compression method id %d", cmid); } @@ -496,9 +538,11 @@ toast_get_compression_handler(struct varlena *attr) static struct varlena * toast_decompress_datum(struct varlena *attr) { - const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + int32 header_size; + const CompressionAmRoutine *cmroutine = + toast_get_compression_handler(attr, &header_size); - return cmroutine->datum_decompress(attr); + return cmroutine->datum_decompress(attr, header_size); } @@ -512,16 +556,19 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + int32 header_size; + const CompressionAmRoutine *cmroutine = + toast_get_compression_handler(attr, &header_size); /* * If the handler supports the slice decompression then decompress the * slice otherwise decompress complete data. */ if (cmroutine->datum_decompress_slice) - return cmroutine->datum_decompress_slice(attr, slicelength); + return cmroutine->datum_decompress_slice(attr, header_size, + slicelength); else - return cmroutine->datum_decompress(attr); + return cmroutine->datum_decompress(attr, header_size); } /* ---------- diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index b04c5a5eb8..a3539065d3 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid) { struct varlena *tmp = NULL; int32 valsize; + bool isCustomCompression = false; const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); @@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid) cmroutine = &lz4_compress_methods; break; default: - elog(ERROR, "Invalid compression method oid %u", cmoid); + isCustomCompression = true; + cmroutine = GetCompressionAmRoutineByAmId(cmoid); + break; } /* Call the actual compression function */ - tmp = cmroutine->datum_compress((const struct varlena *) value); + tmp = cmroutine->datum_compress((const struct varlena *)value, + isCustomCompression ? + TOAST_CUSTOM_COMPRESS_HDRSZ : + TOAST_COMPRESS_HDRSZ); if (!tmp) return PointerGetDatum(NULL); @@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid) if (VARSIZE(tmp) < valsize - 2) { /* successful compression */ - TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ? + CUSTOM_COMPRESSION_ID : + CompressionOidToId(cmoid)); + + /* For custom compression, set the oid of the compression method */ + if (isCustomCompression) + TOAST_COMPRESS_SET_CMOID(tmp, cmoid); + return PointerGetDatum(tmp); } else diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c index e8b035d138..34b0dc1002 100644 --- a/src/backend/access/compression/compress_lz4.c +++ b/src/backend/access/compression/compress_lz4.c @@ -30,7 +30,7 @@ * compressed varlena, or NULL if compression fails. */ static struct varlena * -lz4_cmcompress(const struct varlena *value) +lz4_cmcompress(const struct varlena *value, int32 header_size) { #ifndef HAVE_LIBLZ4 ereport(ERROR, @@ -45,10 +45,10 @@ lz4_cmcompress(const struct varlena *value) valsize = VARSIZE_ANY_EXHDR(value); max_size = LZ4_compressBound(VARSIZE_ANY_EXHDR(value)); - tmp = (struct varlena *) palloc(max_size + TOAST_COMPRESS_HDRSZ); + tmp = (struct varlena *) palloc(max_size + header_size); len = LZ4_compress_default(VARDATA_ANY(value), - (char *) tmp + TOAST_COMPRESS_HDRSZ, + (char *) tmp + header_size, valsize, max_size); if (len <= 0) { @@ -56,7 +56,7 @@ lz4_cmcompress(const struct varlena *value) elog(ERROR, "lz4: could not compress data"); } - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + SET_VARSIZE_COMPRESSED(tmp, len + header_size); return tmp; #endif @@ -68,7 +68,7 @@ lz4_cmcompress(const struct varlena *value) * Returns the decompressed varlena. */ static struct varlena * -lz4_cmdecompress(const struct varlena *value) +lz4_cmdecompress(const struct varlena *value, int32 header_size) { #ifndef HAVE_LIBLZ4 ereport(ERROR, @@ -81,9 +81,9 @@ lz4_cmdecompress(const struct varlena *value) result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); - rawsize = LZ4_decompress_safe(TOAST_COMPRESS_RAWDATA(value), + rawsize = LZ4_decompress_safe((char *) value + header_size, VARDATA(result), - VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + VARSIZE(value) - header_size, TOAST_COMPRESS_RAWSIZE(value)); if (rawsize < 0) elog(ERROR, "lz4: compressed data is corrupted"); @@ -100,7 +100,8 @@ lz4_cmdecompress(const struct varlena *value) * Decompresses part of the data. Returns the decompressed varlena. */ static struct varlena * -lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +lz4_cmdecompress_slice(const struct varlena *value, int32 header_size, + int32 slicelength) { #ifndef HAVE_LIBLZ4 ereport(ERROR, @@ -112,9 +113,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); - rawsize = LZ4_decompress_safe_partial(TOAST_COMPRESS_RAWDATA(value), + rawsize = LZ4_decompress_safe_partial((char *) value + header_size, VARDATA(result), - VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + VARSIZE(value) - header_size, slicelength, TOAST_COMPRESS_RAWSIZE(value)); if (rawsize < 0) diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c index 578be6f1dd..d33ea3d2cf 100644 --- a/src/backend/access/compression/compress_pglz.c +++ b/src/backend/access/compression/compress_pglz.c @@ -27,7 +27,7 @@ * compressed varlena, or NULL if compression fails. */ static struct varlena * -pglz_cmcompress(const struct varlena *value) +pglz_cmcompress(const struct varlena *value, int32 header_size) { int32 valsize, len; @@ -44,16 +44,16 @@ pglz_cmcompress(const struct varlena *value) return NULL; tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + header_size); len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), valsize, - TOAST_COMPRESS_RAWDATA(tmp), + (char *) tmp + header_size, NULL); if (len >= 0) { - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + SET_VARSIZE_COMPRESSED(tmp, len + header_size); return tmp; } @@ -68,7 +68,7 @@ pglz_cmcompress(const struct varlena *value) * Returns the decompressed varlena. */ static struct varlena * -pglz_cmdecompress(const struct varlena *value) +pglz_cmdecompress(const struct varlena *value, int32 header_size) { struct varlena *result; int32 rawsize; @@ -76,8 +76,8 @@ pglz_cmdecompress(const struct varlena *value) result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), - TOAST_COMPRESS_SIZE(value), + rawsize = pglz_decompress((char *) value + header_size, + VARSIZE(value) - header_size, VARDATA(result), TOAST_COMPRESS_RAWSIZE(value), true); @@ -95,7 +95,7 @@ pglz_cmdecompress(const struct varlena *value) * Decompresses part of the data. Returns the decompressed varlena. */ static struct varlena * -pglz_cmdecompress_slice(const struct varlena *value, +pglz_cmdecompress_slice(const struct varlena *value, int32 header_size, int32 slicelength) { struct varlena *result; @@ -103,8 +103,8 @@ pglz_cmdecompress_slice(const struct varlena *value, result = (struct varlena *) palloc(slicelength + VARHDRSZ); - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), - VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + rawsize = pglz_decompress((char *) value + header_size, + VARSIZE(value) - header_size, VARDATA(result), slicelength, false); diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c index e8bfb23ec3..19780f57e7 100644 --- a/src/backend/access/compression/compressamapi.c +++ b/src/backend/access/compression/compressamapi.c @@ -19,6 +19,7 @@ #include "access/htup_details.h" #include "access/reloptions.h" #include "access/table.h" +#include "catalog/pg_am.h" #include "commands/defrem.h" #include "utils/fmgroids.h" #include "utils/syscache.h" @@ -63,6 +64,32 @@ CompressionIdToOid(CompressionId cmid) } } +/* + * GetCompressionAmRoutineByAmId - look up the handler of the compression access + * method with the given OID, and get its CompressionAmRoutine struct. + */ +CompressionAmRoutine * +GetCompressionAmRoutineByAmId(Oid amoid) +{ + regproc amhandler; + Datum datum; + CompressionAmRoutine *routine; + + /* Get handler function OID for the access method */ + amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false); + Assert(OidIsValid(amhandler)); + + /* And finally, call the handler function to get the API struct */ + datum = OidFunctionCall0(amhandler); + routine = (CompressionAmRoutine *) DatumGetPointer(datum); + + if (routine == NULL || !IsA(routine, CompressionAmRoutine)) + elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct", + amhandler); + + return routine; +} + /* check_hook: validate new default_toast_compression */ bool check_default_toast_compression(char **newval, void **extra, GucSource source) diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c index d30bc43514..89a4b01d78 100644 --- a/src/backend/access/index/amapi.c +++ b/src/backend/access/index/amapi.c @@ -46,14 +46,14 @@ GetIndexAmRoutine(Oid amhandler) } /* - * GetIndexAmRoutineByAmId - look up the handler of the index access method - * with the given OID, and get its IndexAmRoutine struct. + * GetAmHandlerByAmId - look up the handler of the index/compression access + * method with the given OID, and get its handler function. * - * If the given OID isn't a valid index access method, returns NULL if - * noerror is true, else throws error. + * If the given OID isn't a valid index/compression access method, returns + * Invalid Oid if noerror is true, else throws error. */ -IndexAmRoutine * -GetIndexAmRoutineByAmId(Oid amoid, bool noerror) +regproc +GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror) { HeapTuple tuple; Form_pg_am amform; @@ -64,24 +64,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror) if (!HeapTupleIsValid(tuple)) { if (noerror) - return NULL; + return InvalidOid; elog(ERROR, "cache lookup failed for access method %u", amoid); } amform = (Form_pg_am) GETSTRUCT(tuple); /* Check if it's an index access method as opposed to some other AM */ - if (amform->amtype != AMTYPE_INDEX) + if (amform->amtype != amtype) { if (noerror) { ReleaseSysCache(tuple); - return NULL; + return InvalidOid; } ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("access method \"%s\" is not of type %s", - NameStr(amform->amname), "INDEX"))); + NameStr(amform->amname), (amtype == AMTYPE_INDEX ? + "INDEX" : "COMPRESSION")))); } amhandler = amform->amhandler; @@ -92,16 +93,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror) if (noerror) { ReleaseSysCache(tuple); - return NULL; + return InvalidOid; } ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("index access method \"%s\" does not have a handler", + errmsg("access method \"%s\" does not have a handler", NameStr(amform->amname)))); } ReleaseSysCache(tuple); + return amhandler; +} + +/* + * GetIndexAmRoutineByAmId - look up the handler of the index access method + * with the given OID, and get its IndexAmRoutine struct. + * + * If the given OID isn't a valid index access method, returns NULL if + * noerror is true, else throws error. + */ +IndexAmRoutine * +GetIndexAmRoutineByAmId(Oid amoid, bool noerror) +{ + regproc amhandler; + + /* Get handler function OID for the access method */ + amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror); + + /* Complain if handler OID is invalid */ + if (!OidIsValid(amhandler)) + { + Assert(noerror); + return NULL; + } + /* And finally, call the handler function to get the API struct. */ return GetIndexAmRoutine(amhandler); } diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index 668e43faed..31d4a3e958 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -241,6 +241,8 @@ get_am_type_string(char amtype) return "INDEX"; case AMTYPE_TABLE: return "TABLE"; + case AMTYPE_COMPRESSION: + return "TABLE"; default: /* shouldn't happen */ elog(ERROR, "invalid access method type '%c'", amtype); @@ -278,6 +280,9 @@ lookup_am_handler_func(List *handler_name, char amtype) case AMTYPE_TABLE: expectedType = TABLE_AM_HANDLEROID; break; + case AMTYPE_COMPRESSION: + expectedType = COMPRESSION_AM_HANDLEROID; + break; default: elog(ERROR, "unrecognized access method type \"%c\"", amtype); } diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 916f6541c1..a63737213d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2089,7 +2089,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot, * compression method is not supported by the target attribute then * we need to decompress it. */ - cmoid = CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value)); + cmoid = toast_get_compression_oid(new_value); if (!IsCompressionSupported(&targetTupDesc->attrs[i], cmoid)) { new_value = detoast_attr(new_value); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 332e60f170..0a0ea0562a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -5304,6 +5304,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name am_type: INDEX { $$ = AMTYPE_INDEX; } | TABLE { $$ = AMTYPE_TABLE; } + | COMPRESSION { $$ = AMTYPE_COMPRESSION; } ; /***************************************************************************** diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index aff5b7e95e..ebd797158a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -5309,7 +5309,6 @@ Datum pg_column_compression(PG_FUNCTION_ARGS) { Datum value = PG_GETARG_DATUM(0); - char *compression; int typlen; struct varlena *varvalue; @@ -5339,10 +5338,8 @@ pg_column_compression(PG_FUNCTION_ARGS) varvalue = (struct varlena *) DatumGetPointer(value); - compression = - get_am_name(CompressionIdToOid(TOAST_COMPRESS_METHOD(varvalue))); - - PG_RETURN_TEXT_P(cstring_to_text(compression)); + PG_RETURN_TEXT_P(cstring_to_text(get_am_name( + toast_get_compression_oid(varvalue)))); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index c09f3972c6..3a79754550 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -13124,6 +13124,9 @@ dumpAccessMethod(Archive *fout, AccessMethodInfo *aminfo) case AMTYPE_TABLE: appendPQExpBufferStr(q, "TYPE TABLE "); break; + case AMTYPE_COMPRESSION: + appendPQExpBufferStr(q, "TYPE COMPRESSION "); + break; default: pg_log_warning("invalid type \"%c\" of access method \"%s\"", aminfo->amtype, qamname); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index c40d498a30..5de2a6a18b 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = { " WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\ " amtype=" CppAsString2(AMTYPE_TABLE) +#define Query_for_list_of_compression_access_methods \ +" SELECT pg_catalog.quote_ident(amname) "\ +" FROM pg_catalog.pg_am "\ +" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\ +" amtype=" CppAsString2(AMTYPE_COMPRESSION) + /* the silly-looking length condition is just to eat up the current word */ #define Query_for_list_of_arguments \ "SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\ diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index d357ebb559..0f7b5b0794 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -285,6 +285,7 @@ typedef struct IndexAmRoutine /* Functions in access/index/amapi.c */ extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler); +extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror); extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror); #endif /* AMAPI_H */ diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h index be5b4da7b8..36c9f57628 100644 --- a/src/include/access/compressamapi.h +++ b/src/include/access/compressamapi.h @@ -30,7 +30,9 @@ typedef enum CompressionId { PGLZ_COMPRESSION_ID = 0, - LZ4_COMPRESSION_ID = 1 + LZ4_COMPRESSION_ID = 1, + /* one free slot for the future built-in method */ + CUSTOM_COMPRESSION_ID = 3 } CompressionId; #define DEFAULT_TOAST_COMPRESSION "pglz" @@ -38,13 +40,18 @@ typedef enum CompressionId extern char *default_toast_compression; /* Use default compression method if it is not specified. */ +#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID) #define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ (storage) != TYPSTORAGE_EXTERNAL) /* compression handler routines */ -typedef struct varlena *(*cmcompress_function) (const struct varlena *value); -typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmcompress_function) (const struct varlena *value, + int32 toast_header_size); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value, + int32 toast_header_size); typedef struct varlena *(*cmdecompress_slice_function) - (const struct varlena *value, int32 slicelength); + (const struct varlena *value, + int32 toast_header_size, + int32 slicelength); /* * API struct for a compression AM. @@ -68,6 +75,7 @@ extern const CompressionAmRoutine lz4_compress_methods; /* access/compression/compressamapi.c */ extern CompressionId CompressionOidToId(Oid cmoid); extern Oid CompressionIdToOid(CompressionId cmid); +extern CompressionAmRoutine *GetCompressionAmRoutineByAmId(Oid amoid); extern void assign_default_toast_compression(const char *newval, void **extra); extern bool check_default_toast_compression(char **newval, void **extra, GucSource source); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index 84ba303e7e..c158354272 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -27,6 +27,17 @@ typedef struct toast_compress_header * rawsize */ } toast_compress_header; +/* + * If the compression method were used, then data also contains + * Oid of compression options + */ +typedef struct toast_compress_header_custom +{ + int32 vl_len_; /* varlena header (do not touch directly!) */ + uint32 info; /* 2 bits for compression method + rawsize */ + Oid cmoid; /* Oid from pg_am */ +} toast_compress_header_custom; + #define RAWSIZEMASK (0x3FFFFFFFU) /* @@ -38,6 +49,7 @@ typedef struct toast_compress_header * two highest bits. */ #define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) +#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom)) #define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->info & RAWSIZEMASK) #define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> 30) #define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) @@ -49,6 +61,9 @@ typedef struct toast_compress_header ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << 30); \ } while (0) +#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \ + (((toast_compress_header_custom *)(ptr))->cmoid = (oid)) + extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 151478bcdd..636ab4021b 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -235,13 +235,51 @@ SELECT pg_column_compression(f1) FROM cmdata; pglz (2 rows) +-- create compression method +CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler; +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL; +INSERT INTO cmdata VALUES (repeat('1234567890',1004)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz2 | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + lz4 + pglz + pglz2 +(3 rows) + +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2); +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + lz4 + lz4 + pglz2 +(3 rows) + +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | +Indexes: + "idx" btree (f1) + -- check data is ok SELECT length(f1) FROM cmdata; length -------- 10000 10040 -(2 rows) + 10040 +(3 rows) SELECT length(f1) FROM cmdata1; length diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index 81182fe949..3177ff91e8 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -232,13 +232,52 @@ SELECT pg_column_compression(f1) FROM cmdata; pglz (2 rows) +-- create compression method +CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler; +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL; +INSERT INTO cmdata VALUES (repeat('1234567890',1004)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz2 | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz + pglz + pglz2 +(3 rows) + +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2); +ERROR: not built with lz4 support +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz + pglz + pglz2 +(3 rows) + +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz2 | | +Indexes: + "idx" btree (f1) + -- check data is ok SELECT length(f1) FROM cmdata; length -------- 10000 10040 -(2 rows) + 10040 +(3 rows) SELECT length(f1) FROM cmdata1; length diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index e447316573..0552eeca16 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -95,6 +95,16 @@ SELECT pg_column_compression(f1) FROM cmdata; ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL; SELECT pg_column_compression(f1) FROM cmdata; +-- create compression method +CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler; +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL; +INSERT INTO cmdata VALUES (repeat('1234567890',1004)); +\d+ cmdata +SELECT pg_column_compression(f1) FROM cmdata; +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2); +SELECT pg_column_compression(f1) FROM cmdata; +\d+ cmdata + -- check data is ok SELECT length(f1) FROM cmdata; SELECT length(f1) FROM cmdata1; -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0006-new-compression-method-extension-for-zlib.patch" From cca457e70e163c27b02c3d1374a57030771442c6 Mon Sep 17 00:00:00 2001 From: dilipkumar Date: Fri, 16 Oct 2020 19:56:14 +0530 Subject: [PATCH v21 6/7] new compression method extension for zlib Dilip Kumar --- contrib/Makefile | 1 + contrib/cmzlib/.gitignore | 4 + contrib/cmzlib/Makefile | 26 +++++ contrib/cmzlib/cmzlib--1.0.sql | 13 +++ contrib/cmzlib/cmzlib.c | 157 +++++++++++++++++++++++++++++ contrib/cmzlib/cmzlib.control | 5 + contrib/cmzlib/expected/cmzlib.out | 53 ++++++++++ contrib/cmzlib/sql/cmzlib.sql | 22 ++++ 8 files changed, 281 insertions(+) create mode 100644 contrib/cmzlib/.gitignore create mode 100644 contrib/cmzlib/Makefile create mode 100644 contrib/cmzlib/cmzlib--1.0.sql create mode 100644 contrib/cmzlib/cmzlib.c create mode 100644 contrib/cmzlib/cmzlib.control create mode 100644 contrib/cmzlib/expected/cmzlib.out create mode 100644 contrib/cmzlib/sql/cmzlib.sql diff --git a/contrib/Makefile b/contrib/Makefile index cdc041c7db..6c61872d94 100644 --- a/contrib/Makefile +++ b/contrib/Makefile @@ -12,6 +12,7 @@ SUBDIRS = \ bloom \ btree_gin \ btree_gist \ + cmzlib \ citext \ cube \ dblink \ diff --git a/contrib/cmzlib/.gitignore b/contrib/cmzlib/.gitignore new file mode 100644 index 0000000000..5dcb3ff972 --- /dev/null +++ b/contrib/cmzlib/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/contrib/cmzlib/Makefile b/contrib/cmzlib/Makefile new file mode 100644 index 0000000000..956fbe7cc8 --- /dev/null +++ b/contrib/cmzlib/Makefile @@ -0,0 +1,26 @@ +# contrib/cmzlib/Makefile + +MODULE_big = cmzlib +OBJS = \ + $(WIN32RES) \ + cmzlib.o + +EXTENSION = cmzlib +DATA = cmzlib--1.0.sql +PGFILEDESC = "zlib compression method " + +SHLIB_LINK += $(filter -lz, $(LIBS)) + +REGRESS = cmzlib + + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = contrib/cmzlib +top_builddir = ../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/contrib/cmzlib/cmzlib--1.0.sql b/contrib/cmzlib/cmzlib--1.0.sql new file mode 100644 index 0000000000..41f2f95870 --- /dev/null +++ b/contrib/cmzlib/cmzlib--1.0.sql @@ -0,0 +1,13 @@ +/* contrib/cm_lz4/cmzlib--1.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION cmzlib" to load this file. \quit + +CREATE FUNCTION zlibhandler(internal) +RETURNS compression_am_handler +AS 'MODULE_PATHNAME' +LANGUAGE C; + +-- Compression method +CREATE ACCESS METHOD zlib TYPE COMPRESSION HANDLER zlibhandler; +COMMENT ON ACCESS METHOD zlib IS 'zlib compression method'; diff --git a/contrib/cmzlib/cmzlib.c b/contrib/cmzlib/cmzlib.c new file mode 100644 index 0000000000..686a7c7e0d --- /dev/null +++ b/contrib/cmzlib/cmzlib.c @@ -0,0 +1,157 @@ +/*------------------------------------------------------------------------- + * + * cmzlib.c + * zlib compression method + * + * Copyright (c) 2015-2018, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * contrib/cmzlib/cmzlib.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" +#include "access/compressamapi.h" +#include "access/toast_internals.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +#include + +PG_MODULE_MAGIC; + +PG_FUNCTION_INFO_V1(zlibhandler); + +void _PG_init(void); + +/* + * Module initialize function: initialize info about zlib + */ +void +_PG_init(void) +{ + +} + +#define ZLIB_MAX_DICTIONARY_LENGTH 32768 +#define ZLIB_DICTIONARY_DELIM (" ,") + +typedef struct +{ + int level; + Bytef dict[ZLIB_MAX_DICTIONARY_LENGTH]; + unsigned int dictlen; +} zlib_state; + +/* + * zlib_cmcompress - compression routine for zlib compression method + * + * Compresses source into dest using the default compression level. + * Returns the compressed varlena, or NULL if compression fails. + */ +static struct varlena * +zlib_cmcompress(const struct varlena *value, int32 header_size) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + z_streamp zp; + int res; + zlib_state state; + + state.level = Z_DEFAULT_COMPRESSION; + + zp = (z_streamp) palloc(sizeof(z_stream)); + zp->zalloc = Z_NULL; + zp->zfree = Z_NULL; + zp->opaque = Z_NULL; + + if (deflateInit(zp, state.level) != Z_OK) + elog(ERROR, "could not initialize compression library: %s", zp->msg); + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + tmp = (struct varlena *) palloc(valsize + header_size); + zp->next_in = (void *) VARDATA_ANY(value); + zp->avail_in = valsize; + zp->avail_out = valsize; + zp->next_out = (void *) ((char *) tmp + header_size); + + do + { + res = deflate(zp, Z_FINISH); + if (res == Z_STREAM_ERROR) + elog(ERROR, "could not compress data: %s", zp->msg); + } while (zp->avail_in != 0); + + Assert(res == Z_STREAM_END); + + len = valsize - zp->avail_out; + if (deflateEnd(zp) != Z_OK) + elog(ERROR, "could not close compression stream: %s", zp->msg); + pfree(zp); + + if (len > 0) + { + SET_VARSIZE_COMPRESSED(tmp, len + header_size); + return tmp; + } + + pfree(tmp); + return NULL; +} + +/* + * zlib_cmdecompress - decompression routine for zlib compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +zlib_cmdecompress(const struct varlena *value, int32 header_size) +{ + struct varlena *result; + z_streamp zp; + int res = Z_OK; + + zp = (z_streamp) palloc(sizeof(z_stream)); + zp->zalloc = Z_NULL; + zp->zfree = Z_NULL; + zp->opaque = Z_NULL; + + if (inflateInit(zp) != Z_OK) + elog(ERROR, "could not initialize compression library: %s", zp->msg); + + zp->next_in = (void *) ((char *) value + header_size); + zp->avail_in = VARSIZE(value) - header_size; + zp->avail_out = VARRAWSIZE_4B_C(value); + + result = (struct varlena *) palloc(zp->avail_out + VARHDRSZ); + SET_VARSIZE(result, zp->avail_out + VARHDRSZ); + zp->next_out = (void *) VARDATA(result); + + while (zp->avail_in > 0) + { + res = inflate(zp, 0); + if (!(res == Z_OK || res == Z_STREAM_END)) + elog(ERROR, "could not uncompress data: %s", zp->msg); + } + + if (inflateEnd(zp) != Z_OK) + elog(ERROR, "could not close compression library: %s", zp->msg); + + pfree(zp); + return result; +} + +const CompressionAmRoutine zlib_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = zlib_cmcompress, + .datum_decompress = zlib_cmdecompress, + .datum_decompress_slice = NULL}; + +Datum +zlibhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&zlib_compress_methods); +} diff --git a/contrib/cmzlib/cmzlib.control b/contrib/cmzlib/cmzlib.control new file mode 100644 index 0000000000..2eb10f3a83 --- /dev/null +++ b/contrib/cmzlib/cmzlib.control @@ -0,0 +1,5 @@ +# cm_lz4 extension +comment = 'cmzlib compression method' +default_version = '1.0' +module_pathname = '$libdir/cmzlib' +relocatable = true diff --git a/contrib/cmzlib/expected/cmzlib.out b/contrib/cmzlib/expected/cmzlib.out new file mode 100644 index 0000000000..2b6fac7e0b --- /dev/null +++ b/contrib/cmzlib/expected/cmzlib.out @@ -0,0 +1,53 @@ +CREATE EXTENSION cmzlib; +-- zlib compression +CREATE TABLE zlibtest(f1 TEXT COMPRESSION pglz); +INSERT INTO zlibtest VALUES(repeat('1234567890',1004)); +INSERT INTO zlibtest VALUES(repeat('1234567890 one two three',1004)); +SELECT length(f1) FROM zlibtest; + length +-------- + 10040 + 24096 +(2 rows) + +-- alter compression method with rewrite +ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz; +\d+ zlibtest + Table "public.zlibtest" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | + +ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION zlib; +\d+ zlibtest + Table "public.zlibtest" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | zlib | | + +-- preserve old compression method +ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (zlib); +INSERT INTO zlibtest VALUES (repeat('1234567890',1004)); +\d+ zlibtest + Table "public.zlibtest" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | + +SELECT pg_column_compression(f1) FROM zlibtest; + pg_column_compression +----------------------- + zlib + zlib + pglz +(3 rows) + +SELECT length(f1) FROM zlibtest; + length +-------- + 10040 + 24096 + 10040 +(3 rows) + +DROP TABLE zlibtest; diff --git a/contrib/cmzlib/sql/cmzlib.sql b/contrib/cmzlib/sql/cmzlib.sql new file mode 100644 index 0000000000..ea8d206625 --- /dev/null +++ b/contrib/cmzlib/sql/cmzlib.sql @@ -0,0 +1,22 @@ +CREATE EXTENSION cmzlib; + +-- zlib compression +CREATE TABLE zlibtest(f1 TEXT COMPRESSION pglz); +INSERT INTO zlibtest VALUES(repeat('1234567890',1004)); +INSERT INTO zlibtest VALUES(repeat('1234567890 one two three',1004)); +SELECT length(f1) FROM zlibtest; + +-- alter compression method with rewrite +ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz; +\d+ zlibtest +ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION zlib; +\d+ zlibtest + +-- preserve old compression method +ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (zlib); +INSERT INTO zlibtest VALUES (repeat('1234567890',1004)); +\d+ zlibtest +SELECT pg_column_compression(f1) FROM zlibtest; +SELECT length(f1) FROM zlibtest; + +DROP TABLE zlibtest; -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0007-Support-compression-methods-options.patch" From 172956bacd687fdeb6628d74b00efa5d17c642ef Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Fri, 29 Jan 2021 15:46:33 +0530 Subject: [PATCH v21 7/7] Support compression methods options Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Tomas Vondra and Robert Haas Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby --- contrib/cmzlib/cmzlib.c | 75 +++++++++++-- doc/src/sgml/ref/alter_table.sgml | 6 +- doc/src/sgml/ref/create_table.sgml | 8 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/reloptions.c | 64 +++++++++++ src/backend/access/common/toast_internals.c | 14 ++- src/backend/access/compression/compress_lz4.c | 69 +++++++++++- .../access/compression/compress_pglz.c | 98 +++++++++++++++-- src/backend/access/table/toast_helper.c | 8 +- src/backend/bootstrap/bootparse.y | 1 + src/backend/catalog/heap.c | 15 ++- src/backend/catalog/index.c | 43 ++++++-- src/backend/catalog/toasting.c | 1 + src/backend/commands/cluster.c | 1 + src/backend/commands/compressioncmds.c | 79 +++++++++++++- src/backend/commands/foreigncmds.c | 44 -------- src/backend/commands/tablecmds.c | 102 +++++++++++++----- src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 16 ++- src/backend/parser/parse_utilcmd.c | 2 +- src/bin/pg_dump/pg_dump.c | 23 +++- src/bin/pg_dump/pg_dump.h | 2 + src/include/access/compressamapi.h | 20 +++- src/include/access/toast_helper.h | 2 + src/include/access/toast_internals.h | 2 +- src/include/catalog/heap.h | 2 + src/include/catalog/pg_attribute.h | 3 + src/include/commands/defrem.h | 9 +- src/include/nodes/parsenodes.h | 1 + src/test/regress/expected/compression.out | 52 +++++++++ src/test/regress/expected/compression_1.out | 55 ++++++++++ src/test/regress/expected/misc_sanity.out | 3 +- src/test/regress/sql/compression.sql | 18 ++++ 36 files changed, 717 insertions(+), 133 deletions(-) diff --git a/contrib/cmzlib/cmzlib.c b/contrib/cmzlib/cmzlib.c index 686a7c7e0d..d8c2865f21 100644 --- a/contrib/cmzlib/cmzlib.c +++ b/contrib/cmzlib/cmzlib.c @@ -14,6 +14,7 @@ #include "postgres.h" #include "access/compressamapi.h" #include "access/toast_internals.h" +#include "commands/defrem.h" #include "fmgr.h" #include "utils/builtins.h" @@ -45,6 +46,62 @@ typedef struct unsigned int dictlen; } zlib_state; +/* + * Check options if specified. All validation is located here so + * we don't need to do it again in cminitstate function. + */ +static void +zlib_cmcheck(List *options) +{ + ListCell *lc; + + foreach(lc, options) + { + DefElem *def = (DefElem *) lfirst(lc); + + if (strcmp(def->defname, "level") == 0) + { + int8 level = pg_atoi(defGetString(def), sizeof(int8), 0); + + if (level < 0 || level > 9) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unexpected value for zlib compression level: \"%s\"", + defGetString(def)), + errhint("expected value between 0 and 9") + )); + } + else + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_PARAMETER), + errmsg("unknown compression option for zlib: \"%s\"", def->defname))); + } +} + +static void * +zlib_cminitstate(List *options) +{ + zlib_state *state = NULL; + + state = palloc0(sizeof(zlib_state)); + state->level = Z_DEFAULT_COMPRESSION; + + if (list_length(options) > 0) + { + ListCell *lc; + + foreach(lc, options) + { + DefElem *def = (DefElem *) lfirst(lc); + + if (strcmp(def->defname, "level") == 0) + state->level = pg_atoi(defGetString(def), sizeof(int), 0); + } + } + + return state; +} + /* * zlib_cmcompress - compression routine for zlib compression method * @@ -52,23 +109,21 @@ typedef struct * Returns the compressed varlena, or NULL if compression fails. */ static struct varlena * -zlib_cmcompress(const struct varlena *value, int32 header_size) +zlib_cmcompress(const struct varlena *value, int32 header_size, void *options) { - int32 valsize, - len; + int32 valsize, + len; struct varlena *tmp = NULL; - z_streamp zp; - int res; - zlib_state state; - - state.level = Z_DEFAULT_COMPRESSION; + z_streamp zp; + int res; + zlib_state *state = (zlib_state *) options; zp = (z_streamp) palloc(sizeof(z_stream)); zp->zalloc = Z_NULL; zp->zfree = Z_NULL; zp->opaque = Z_NULL; - if (deflateInit(zp, state.level) != Z_OK) + if (deflateInit(zp, state->level) != Z_OK) elog(ERROR, "could not initialize compression library: %s", zp->msg); valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); @@ -146,6 +201,8 @@ zlib_cmdecompress(const struct varlena *value, int32 header_size) const CompressionAmRoutine zlib_compress_methods = { .type = T_CompressionAmRoutine, + .datum_check = zlib_cmcheck, + .datum_initstate = zlib_cminitstate, .datum_compress = zlib_cmcompress, .datum_decompress = zlib_cmdecompress, .datum_decompress_slice = NULL}; diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index 46f34bffe5..e496f02cdc 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] name ALTER [ COLUMN ] column_name SET ( attribute_option = value [, ... ] ) ALTER [ COLUMN ] column_name RESET ( attribute_option [, ... ] ) ALTER [ COLUMN ] column_name SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN } - ALTER [ COLUMN ] column_name SET COMPRESSION compression_method [ PRESERVE (compression_preserve_list) | PRESERVE ALL ] + ALTER [ COLUMN ] column_name SET COMPRESSION compression_method [ WITH (compression_method_options) ] [ PRESERVE (compression_preserve_list) | PRESERVE ALL ] ADD table_constraint [ NOT VALID ] ADD table_constraint_using_index ALTER CONSTRAINT constraint_name [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -393,7 +393,9 @@ WITH ( MODULUS numeric_literal, REM This clause adds compression to a column. The Compression method could be created with or it can be set to any available compression method. The built-in methods - are pglz and lz4. + are pglz and lz4. If the + compression method has options they can be specified with the + WITH parameter. The PRESERVE list contains a list of compression methods used in the column and determines which of them may be kept. Without PRESERVE or if any of the pre-existing diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 10b468fb2d..e5514bbd1b 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,7 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( sequence_options ) ] | UNIQUE index_parameters | PRIMARY KEY index_parameters | - COMPRESSION compression_method | + COMPRESSION compression_method [ WITH (compression_options) ] | REFERENCES reftable [ ( refcolumn ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE referential_action ] [ ON UPDATE referential_action ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -994,7 +994,7 @@ WITH ( MODULUS numeric_literal, REM - COMPRESSION compression_method + COMPRESSION compression_method [ WITH (compression_am_options) ] This clause adds the compression method to a column. The Compression @@ -1002,7 +1002,9 @@ WITH ( MODULUS numeric_literal, REM it can be set from the available compression methods. The built-in methods are pglz and lz4. If no compression method is specified, then compressible types will have - the default compression method pglz. + the default compression method pglz. If the compression + method has options they can be specified with the WITH + parameter. diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index 0ab5712c71..76f824bc6f 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -38,6 +38,7 @@ #include "access/toast_internals.h" #include "access/tupdesc.h" #include "access/tupmacs.h" +#include "commands/defrem.h" #include "utils/datum.h" #include "utils/memutils.h" @@ -215,8 +216,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, { Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, keyno); + List *acoption = GetAttributeCompressionOptions(att); Datum cvalue = toast_compress_datum(value, - att->attcompression); + att->attcompression, + acoption); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index 1d43d5d2ff..0d3307e94f 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -22,6 +22,7 @@ #include "access/htup_details.h" #include "access/itup.h" #include "access/toast_internals.h" +#include "commands/defrem.h" /* * This enables de-toasting of index entries. Needed until VACUUM is @@ -104,8 +105,9 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { + List *acoption = GetAttributeCompressionOptions(att); Datum cvalue = toast_compress_datum(untoasted_values[i], - att->attcompression); + att->attcompression, acoption); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index c687d3ee9e..2dda6c038b 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -2112,3 +2112,67 @@ AlterTableGetRelOptionsLockLevel(List *defList) return lockmode; } + +/* + * Convert a DefElem list to the text array format that is used in + * pg_foreign_data_wrapper, pg_foreign_server, pg_user_mapping, and + * pg_foreign_table. + * + * Returns the array in the form of a Datum, or PointerGetDatum(NULL) + * if the list is empty. + * + * Note: The array is usually stored to database without further + * processing, hence any validation should be done before this + * conversion. + */ +Datum +optionListToArray(List *options) +{ + ArrayBuildState *astate = NULL; + ListCell *cell; + + foreach(cell, options) + { + DefElem *def = lfirst(cell); + const char *value; + Size len; + text *t; + + value = defGetString(def); + len = VARHDRSZ + strlen(def->defname) + 1 + strlen(value); + t = palloc(len + 1); + SET_VARSIZE(t, len); + sprintf(VARDATA(t), "%s=%s", def->defname, value); + + astate = accumArrayResult(astate, PointerGetDatum(t), + false, TEXTOID, + CurrentMemoryContext); + } + + if (astate) + return makeArrayResult(astate, CurrentMemoryContext); + + return PointerGetDatum(NULL); +} + +/* + * Return human readable list of reloptions + */ +char * +formatRelOptions(List *options) +{ + StringInfoData buf; + ListCell *cell; + + initStringInfo(&buf); + + foreach(cell, options) + { + DefElem *def = (DefElem *) lfirst(cell); + + appendStringInfo(&buf, "%s%s=%s", buf.len > 0 ? ", " : "", + def->defname, defGetString(def)); + } + + return buf.data; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index a3539065d3..5ed3312f39 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,10 +44,11 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value, Oid cmoid) +toast_compress_datum(Datum value, Oid cmoid, List *cmoptions) { struct varlena *tmp = NULL; int32 valsize; + void *options = NULL; bool isCustomCompression = false; const CompressionAmRoutine *cmroutine = NULL; @@ -71,11 +72,20 @@ toast_compress_datum(Datum value, Oid cmoid) break; } + if (cmroutine->datum_initstate) + options = cmroutine->datum_initstate(cmoptions); + /* Call the actual compression function */ tmp = cmroutine->datum_compress((const struct varlena *)value, isCustomCompression ? TOAST_CUSTOM_COMPRESS_HDRSZ : - TOAST_COMPRESS_HDRSZ); + TOAST_COMPRESS_HDRSZ, options); + if (options != NULL) + { + pfree(options); + list_free(cmoptions); + } + if (!tmp) return PointerGetDatum(NULL); diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c index 34b0dc1002..375c4e852d 100644 --- a/src/backend/access/compression/compress_lz4.c +++ b/src/backend/access/compression/compress_lz4.c @@ -16,6 +16,7 @@ #include "postgres.h" #include "access/compressamapi.h" #include "access/toast_internals.h" +#include "commands/defrem.h" #include "fmgr.h" #include "utils/builtins.h" @@ -23,6 +24,63 @@ #include "lz4.h" #endif +/* + * Check options if specified. All validation is located here so + * we don't need to do it again in cminitstate function. + */ +static void +lz4_cmcheck(List *options) +{ + ListCell *lc; + + foreach(lc, options) + { + DefElem *def = (DefElem *) lfirst(lc); + + if (strcmp(def->defname, "acceleration") == 0) + { + int32 acceleration = + pg_atoi(defGetString(def), sizeof(acceleration), 0); + + if (acceleration < INT32_MIN || acceleration > INT32_MAX) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unexpected value for lz4 compression acceleration: \"%s\"", + defGetString(def)), + errhint("expected value between INT32_MIN and INT32_MAX") + )); + } + else + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_PARAMETER), + errmsg("unknown compression option for lz4: \"%s\"", def->defname))); + } +} + +static void * +lz4_cminitstate(List *options) +{ + int32 *acceleration = palloc(sizeof(int32)); + + /* initialize with the default acceleration */ + *acceleration = 1; + + if (list_length(options) > 0) + { + ListCell *lc; + + foreach(lc, options) + { + DefElem *def = (DefElem *) lfirst(lc); + + if (strcmp(def->defname, "acceleration") == 0) + *acceleration = pg_atoi(defGetString(def), sizeof(int32), 0); + } + } + + return acceleration; +} + /* * lz4_cmcompress - compression routine for lz4 compression method * @@ -30,7 +88,7 @@ * compressed varlena, or NULL if compression fails. */ static struct varlena * -lz4_cmcompress(const struct varlena *value, int32 header_size) +lz4_cmcompress(const struct varlena *value, int32 header_size, void *options) { #ifndef HAVE_LIBLZ4 ereport(ERROR, @@ -40,6 +98,7 @@ lz4_cmcompress(const struct varlena *value, int32 header_size) int32 valsize; int32 len; int32 max_size; + int32 *acceleration = (int32 *) options; struct varlena *tmp = NULL; valsize = VARSIZE_ANY_EXHDR(value); @@ -47,9 +106,9 @@ lz4_cmcompress(const struct varlena *value, int32 header_size) max_size = LZ4_compressBound(VARSIZE_ANY_EXHDR(value)); tmp = (struct varlena *) palloc(max_size + header_size); - len = LZ4_compress_default(VARDATA_ANY(value), - (char *) tmp + header_size, - valsize, max_size); + len = LZ4_compress_fast(VARDATA_ANY(value), + (char *) tmp + header_size, + valsize, max_size, *acceleration); if (len <= 0) { pfree(tmp); @@ -129,6 +188,8 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 header_size, const CompressionAmRoutine lz4_compress_methods = { .type = T_CompressionAmRoutine, + .datum_check = lz4_cmcheck, + .datum_initstate = lz4_cminitstate, .datum_compress = lz4_cmcompress, .datum_decompress = lz4_cmdecompress, .datum_decompress_slice = lz4_cmdecompress_slice diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c index d33ea3d2cf..0396729c70 100644 --- a/src/backend/access/compression/compress_pglz.c +++ b/src/backend/access/compression/compress_pglz.c @@ -15,11 +15,92 @@ #include "access/compressamapi.h" #include "access/toast_internals.h" +#include "commands/defrem.h" #include "common/pg_lzcompress.h" #include "fmgr.h" #include "utils/builtins.h" +#define PGLZ_OPTIONS_COUNT 6 + +static char *PGLZ_options[PGLZ_OPTIONS_COUNT] = { + "min_input_size", + "max_input_size", + "min_comp_rate", + "first_success_by", + "match_size_good", + "match_size_drop" +}; + +/* + * Convert value from reloptions to int32, and report if it is not correct. + * Also checks parameter names + */ +static int32 +parse_option(char *name, char *value) +{ + int i; + + for (i = 0; i < PGLZ_OPTIONS_COUNT; i++) + { + if (strcmp(PGLZ_options[i], name) == 0) + return pg_atoi(value, 4, 0); + } + + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_PARAMETER), + errmsg("unknown compression option for pglz: \"%s\"", name))); +} + +/* + * Check PGLZ options if specified + */ +static void +pglz_cmcheck(List *options) +{ + ListCell *lc; + + foreach(lc, options) + { + DefElem *def = (DefElem *) lfirst(lc); + + parse_option(def->defname, defGetString(def)); + } +} + +/* + * Configure PGLZ_Strategy struct for compression function + */ +static void * +pglz_cminitstate(List *options) +{ + ListCell *lc; + PGLZ_Strategy *strategy = palloc(sizeof(PGLZ_Strategy)); + + /* initialize with default strategy values */ + memcpy(strategy, PGLZ_strategy_default, sizeof(PGLZ_Strategy)); + foreach(lc, options) + { + DefElem *def = (DefElem *) lfirst(lc); + int32 val = parse_option(def->defname, defGetString(def)); + + /* fill the strategy */ + if (strcmp(def->defname, "min_input_size") == 0) + strategy->min_input_size = val; + else if (strcmp(def->defname, "max_input_size") == 0) + strategy->max_input_size = val; + else if (strcmp(def->defname, "min_comp_rate") == 0) + strategy->min_comp_rate = val; + else if (strcmp(def->defname, "first_success_by") == 0) + strategy->first_success_by = val; + else if (strcmp(def->defname, "match_size_good") == 0) + strategy->match_size_good = val; + else if (strcmp(def->defname, "match_size_drop") == 0) + strategy->match_size_drop = val; + } + return (void *) strategy; +} + /* * pglz_cmcompress - compression routine for pglz compression method * @@ -27,20 +108,22 @@ * compressed varlena, or NULL if compression fails. */ static struct varlena * -pglz_cmcompress(const struct varlena *value, int32 header_size) +pglz_cmcompress(const struct varlena *value, int32 header_size, void *options) { int32 valsize, len; - struct varlena *tmp = NULL; + struct varlena *tmp = NULL; + PGLZ_Strategy *strategy; valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + strategy = (PGLZ_Strategy *) options; /* * No point in wasting a palloc cycle if value size is outside the allowed * range for compression. */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) + if (valsize < strategy->min_input_size || + valsize > strategy->max_input_size) return NULL; tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + @@ -49,7 +132,7 @@ pglz_cmcompress(const struct varlena *value, int32 header_size) len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), valsize, (char *) tmp + header_size, - NULL); + strategy); if (len >= 0) { @@ -118,10 +201,11 @@ pglz_cmdecompress_slice(const struct varlena *value, int32 header_size, const CompressionAmRoutine pglz_compress_methods = { .type = T_CompressionAmRoutine, + .datum_check = pglz_cmcheck, + .datum_initstate = pglz_cminitstate, .datum_compress = pglz_cmcompress, .datum_decompress = pglz_cmdecompress, - .datum_decompress_slice = pglz_cmdecompress_slice -}; + .datum_decompress_slice = pglz_cmdecompress_slice}; /* pglz compression handler function */ Datum diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index 53f78f9c3e..c9d44c62fa 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -15,11 +15,13 @@ #include "postgres.h" #include "access/detoast.h" +#include "access/reloptions.h" #include "access/table.h" #include "access/toast_helper.h" #include "access/toast_internals.h" #include "catalog/pg_type_d.h" - +#include "commands/defrem.h" +#include "utils/syscache.h" /* * Prepare to TOAST a tuple. @@ -55,6 +57,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; ttc->ttc_attr[i].tai_compression = att->attcompression; + ttc->ttc_attr[i].tai_cmoptions = GetAttributeCompressionOptions(att); if (ttc->ttc_oldvalues != NULL) { @@ -230,7 +233,8 @@ toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; - new_value = toast_compress_datum(*value, attr->tai_compression); + new_value = toast_compress_datum(*value, attr->tai_compression, + attr->tai_cmoptions); if (DatumGetPointer(new_value) != NULL) { diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y index 5fcd004e1b..a43e0e197c 100644 --- a/src/backend/bootstrap/bootparse.y +++ b/src/backend/bootstrap/bootparse.y @@ -239,6 +239,7 @@ Boot_CreateStmt: true, false, InvalidOid, + NULL, NULL); elog(DEBUG4, "relation created with OID %u", id); } diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index b53b6b50e6..18b04816f7 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -741,6 +741,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, TupleDesc tupdesc, Oid new_rel_oid, Datum *attoptions, + Datum *attcmoptions, CatalogIndexState indstate) { TupleTableSlot **slot; @@ -796,6 +797,11 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, else slot[slotCount]->tts_isnull[Anum_pg_attribute_attoptions - 1] = true; + if (attcmoptions && attcmoptions[natts] != (Datum) 0) + slot[slotCount]->tts_values[Anum_pg_attribute_attcmoptions - 1] = attcmoptions[natts]; + else + slot[slotCount]->tts_isnull[Anum_pg_attribute_attcmoptions - 1] = true; + /* start out with empty permissions and empty options */ slot[slotCount]->tts_isnull[Anum_pg_attribute_attacl - 1] = true; slot[slotCount]->tts_isnull[Anum_pg_attribute_attfdwoptions - 1] = true; @@ -843,6 +849,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, static void AddNewAttributeTuples(Oid new_rel_oid, TupleDesc tupdesc, + Datum *acoption, char relkind) { Relation rel; @@ -861,7 +868,8 @@ AddNewAttributeTuples(Oid new_rel_oid, /* set stats detail level to a sane default */ for (int i = 0; i < natts; i++) tupdesc->attrs[i].attstattarget = -1; - InsertPgAttributeTuples(rel, tupdesc, new_rel_oid, NULL, indstate); + InsertPgAttributeTuples(rel, tupdesc, new_rel_oid, + NULL, acoption, indstate); /* add dependencies on their datatypes and collations */ for (int i = 0; i < natts; i++) @@ -893,7 +901,7 @@ AddNewAttributeTuples(Oid new_rel_oid, td = CreateTupleDesc(lengthof(SysAtt), (FormData_pg_attribute **) &SysAtt); - InsertPgAttributeTuples(rel, td, new_rel_oid, NULL, indstate); + InsertPgAttributeTuples(rel, td, new_rel_oid, NULL, NULL, indstate); FreeTupleDesc(td); } @@ -1160,6 +1168,7 @@ heap_create_with_catalog(const char *relname, bool allow_system_table_mods, bool is_internal, Oid relrewrite, + Datum *acoptions, ObjectAddress *typaddress) { Relation pg_class_desc; @@ -1418,7 +1427,7 @@ heap_create_with_catalog(const char *relname, /* * now add tuples to pg_attribute for the attributes in our new relation. */ - AddNewAttributeTuples(relid, new_rel_desc->rd_att, relkind); + AddNewAttributeTuples(relid, new_rel_desc->rd_att, acoptions, relkind); /* * Make a dependency link to force the relation to be deleted if its diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e6040923dc..2a39e0ce64 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -107,10 +107,12 @@ static TupleDesc ConstructTupleDescriptor(Relation heapRelation, List *indexColNames, Oid accessMethodObjectId, Oid *collationObjectId, - Oid *classObjectId); + Oid *classObjectId, + Datum *acoptions); static void InitializeAttributeOids(Relation indexRelation, int numatts, Oid indexoid); -static void AppendAttributeTuples(Relation indexRelation, Datum *attopts); +static void AppendAttributeTuples(Relation indexRelation, Datum *attopts, + Datum *attcmopts); static void UpdateIndexRelation(Oid indexoid, Oid heapoid, Oid parentIndexId, IndexInfo *indexInfo, @@ -272,7 +274,8 @@ ConstructTupleDescriptor(Relation heapRelation, List *indexColNames, Oid accessMethodObjectId, Oid *collationObjectId, - Oid *classObjectId) + Oid *classObjectId, + Datum *acoptions) { int numatts = indexInfo->ii_NumIndexAttrs; int numkeyatts = indexInfo->ii_NumIndexKeyAttrs; @@ -333,6 +336,9 @@ ConstructTupleDescriptor(Relation heapRelation, { /* Simple index column */ const FormData_pg_attribute *from; + HeapTuple attr_tuple; + Datum attcmoptions; + bool isNull; Assert(atnum > 0); /* should've been caught above */ @@ -349,6 +355,23 @@ ConstructTupleDescriptor(Relation heapRelation, to->attstorage = from->attstorage; to->attalign = from->attalign; to->attcompression = from->attcompression; + + attr_tuple = SearchSysCache2(ATTNUM, + ObjectIdGetDatum(from->attrelid), + Int16GetDatum(from->attnum)); + if (!HeapTupleIsValid(attr_tuple)) + elog(ERROR, "cache lookup failed for attribute %d of relation %u", + from->attnum, from->attrelid); + + attcmoptions = SysCacheGetAttr(ATTNUM, attr_tuple, + Anum_pg_attribute_attcmoptions, + &isNull); + if (isNull) + acoptions[i] = PointerGetDatum(NULL); + else + acoptions[i] = + PointerGetDatum(DatumGetArrayTypePCopy(attcmoptions)); + ReleaseSysCache(attr_tuple); } else { @@ -489,7 +512,7 @@ InitializeAttributeOids(Relation indexRelation, * ---------------------------------------------------------------- */ static void -AppendAttributeTuples(Relation indexRelation, Datum *attopts) +AppendAttributeTuples(Relation indexRelation, Datum *attopts, Datum *attcmopts) { Relation pg_attribute; CatalogIndexState indstate; @@ -507,7 +530,8 @@ AppendAttributeTuples(Relation indexRelation, Datum *attopts) */ indexTupDesc = RelationGetDescr(indexRelation); - InsertPgAttributeTuples(pg_attribute, indexTupDesc, InvalidOid, attopts, indstate); + InsertPgAttributeTuples(pg_attribute, indexTupDesc, InvalidOid, + attopts, attcmopts, indstate); CatalogCloseIndexes(indstate); @@ -720,6 +744,7 @@ index_create(Relation heapRelation, bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; char relkind; + Datum *acoptions; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -875,6 +900,8 @@ index_create(Relation heapRelation, indexRelationName, RelationGetRelationName(heapRelation)))); } + acoptions = (Datum *) palloc0(sizeof(Datum) * indexInfo->ii_NumIndexAttrs); + /* * construct tuple descriptor for index tuples */ @@ -883,7 +910,8 @@ index_create(Relation heapRelation, indexColNames, accessMethodObjectId, collationObjectId, - classObjectId); + classObjectId, + acoptions); /* * Allocate an OID for the index, unless we were told what to use. @@ -974,7 +1002,8 @@ index_create(Relation heapRelation, /* * append ATTRIBUTE tuples for the index */ - AppendAttributeTuples(indexRelation, indexInfo->ii_OpclassOptions); + AppendAttributeTuples(indexRelation, indexInfo->ii_OpclassOptions, + acoptions); /* ---------------- * update pg_index diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index a549481557..348b7d3a37 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -260,6 +260,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, true, true, InvalidOid, + NULL, NULL); Assert(toast_relid != InvalidOid); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 096a06f7b3..0ad946d802 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -699,6 +699,7 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, char relpersistence, true, true, OIDOldHeap, + NULL, NULL); Assert(OIDNewHeap != InvalidOid); diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c index 50f0f7faaa..d70408c7c2 100644 --- a/src/backend/commands/compressioncmds.c +++ b/src/backend/commands/compressioncmds.c @@ -29,6 +29,7 @@ #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/syscache.h" /* * Get list of all supported compression methods for the given attribute. @@ -181,7 +182,7 @@ BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve) */ Oid GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression, - bool *need_rewrite) + Datum *acoptions, bool *need_rewrite) { Oid cmoid; char typstorage = get_typstorage(att->atttypid); @@ -207,6 +208,20 @@ GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression, cmoid = get_compression_am_oid(compression->cmname, false); + /* if compression options are given then check them */ + if (compression->options) + { + CompressionAmRoutine *routine = GetCompressionAmRoutineByAmId(cmoid); + + /* we need routine only to call cmcheck function */ + if (routine->datum_check != NULL) + routine->datum_check(compression->options); + + *acoptions = optionListToArray(compression->options); + } + else + *acoptions = PointerGetDatum(NULL); + /* * Determine if the column needs rewrite or not. Rewrite conditions: SET * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not @@ -279,15 +294,71 @@ GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression, * Construct ColumnCompression node from the compression method oid. */ ColumnCompression * -MakeColumnCompression(Oid attcompression) +MakeColumnCompression(Form_pg_attribute att) { ColumnCompression *node; - if (!OidIsValid(attcompression)) + if (!OidIsValid(att->attcompression)) return NULL; node = makeNode(ColumnCompression); - node->cmname = get_am_name(attcompression); + node->cmname = get_am_name(att->attcompression); + node->options = GetAttributeCompressionOptions(att); return node; } + +/* + * Fetch atttribute's compression options + */ +List * +GetAttributeCompressionOptions(Form_pg_attribute att) +{ + HeapTuple attr_tuple; + Datum attcmoptions; + List *acoptions; + bool isNull; + + attr_tuple = SearchSysCache2(ATTNUM, + ObjectIdGetDatum(att->attrelid), + Int16GetDatum(att->attnum)); + if (!HeapTupleIsValid(attr_tuple)) + elog(ERROR, "cache lookup failed for attribute %d of relation %u", + att->attnum, att->attrelid); + + attcmoptions = SysCacheGetAttr(ATTNUM, attr_tuple, + Anum_pg_attribute_attcmoptions, + &isNull); + if (isNull) + acoptions = NULL; + else + acoptions = untransformRelOptions(attcmoptions); + + ReleaseSysCache(attr_tuple); + + return acoptions; +} + +/* + * Compare compression options for two columns. + */ +void +CheckCompressionMismatch(ColumnCompression *c1, ColumnCompression *c2, + const char *attributeName) +{ + if (strcmp(c1->cmname, c2->cmname)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", c1->cmname, c2->cmname))); + + if (!equal(c1->options, c2->options)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression options conflict", + attributeName), + errdetail("(%s) versus (%s)", + formatRelOptions(c1->options), + formatRelOptions(c2->options)))); +} diff --git a/src/backend/commands/foreigncmds.c b/src/backend/commands/foreigncmds.c index eb7103fd3b..ae9ae2c51b 100644 --- a/src/backend/commands/foreigncmds.c +++ b/src/backend/commands/foreigncmds.c @@ -49,50 +49,6 @@ typedef struct /* Internal functions */ static void import_error_callback(void *arg); - -/* - * Convert a DefElem list to the text array format that is used in - * pg_foreign_data_wrapper, pg_foreign_server, pg_user_mapping, and - * pg_foreign_table. - * - * Returns the array in the form of a Datum, or PointerGetDatum(NULL) - * if the list is empty. - * - * Note: The array is usually stored to database without further - * processing, hence any validation should be done before this - * conversion. - */ -static Datum -optionListToArray(List *options) -{ - ArrayBuildState *astate = NULL; - ListCell *cell; - - foreach(cell, options) - { - DefElem *def = lfirst(cell); - const char *value; - Size len; - text *t; - - value = defGetString(def); - len = VARHDRSZ + strlen(def->defname) + 1 + strlen(value); - t = palloc(len + 1); - SET_VARSIZE(t, len); - sprintf(VARDATA(t), "%s=%s", def->defname, value); - - astate = accumArrayResult(astate, PointerGetDatum(t), - false, TEXTOID, - CurrentMemoryContext); - } - - if (astate) - return makeArrayResult(astate, CurrentMemoryContext); - - return PointerGetDatum(NULL); -} - - /* * Transform a list of DefElem into text array format. This is substantially * the same thing as optionListToArray(), except we recognize SET/ADD/DROP diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 9d732fc167..93c1374bd4 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -609,6 +609,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, LOCKMODE parentLockmode; const char *accessMethod = NULL; Oid accessMethodId = InvalidOid; + Datum *acoptions; /* * Truncate relname to appropriate length (probably a waste of time, as @@ -813,6 +814,8 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, cookedDefaults = NIL; attnum = 0; + acoptions = (Datum *) palloc0(sizeof(Datum) * descriptor->natts); + foreach(listptr, stmt->tableElts) { ColumnDef *colDef = lfirst(listptr); @@ -866,8 +869,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (relkind == RELKIND_RELATION || relkind == RELKIND_PARTITIONED_TABLE || relkind == RELKIND_MATVIEW) - attr->attcompression = - GetAttributeCompression(attr, colDef->compression, NULL); + attr->attcompression = GetAttributeCompression(attr, + colDef->compression, + &acoptions[attnum - 1], NULL); else attr->attcompression = InvalidOid; } @@ -921,8 +925,11 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, allowSystemTableMods, false, InvalidOid, + acoptions, typaddress); + pfree(acoptions); + /* * We must bump the command counter to make the newly-created relation * tuple visible for opening. @@ -2432,16 +2439,14 @@ MergeAttributes(List *schema, List *supers, char relpersistence, if (OidIsValid(attribute->attcompression)) { ColumnCompression *compression = - MakeColumnCompression(attribute->attcompression); + MakeColumnCompression(attribute); if (!def->compression) def->compression = compression; - else if (strcmp(def->compression->cmname, compression->cmname)) - ereport(ERROR, - (errcode(ERRCODE_DATATYPE_MISMATCH), - errmsg("column \"%s\" has a compression method conflict", - attributeName), - errdetail("%s versus %s", def->compression->cmname, compression->cmname))); + else + CheckCompressionMismatch(def->compression, + compression, + attributeName); } def->inhcount++; @@ -2478,8 +2483,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; - def->compression = MakeColumnCompression( - attribute->attcompression); + def->compression = MakeColumnCompression(attribute); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2729,14 +2733,9 @@ MergeAttributes(List *schema, List *supers, char relpersistence, if (!def->compression) def->compression = newdef->compression; else if (newdef->compression) - { - if (strcmp(def->compression->cmname, newdef->compression->cmname)) - ereport(ERROR, - (errcode(ERRCODE_DATATYPE_MISMATCH), - errmsg("column \"%s\" has a compression method conflict", - attributeName), - errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname))); - } + CheckCompressionMismatch(def->compression, + newdef->compression, + attributeName); /* Mark the column as locally defined */ def->is_local = true; @@ -6259,6 +6258,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, AclResult aclresult; ObjectAddress address; TupleDesc tupdesc; + Datum acoptions = PointerGetDatum(NULL); FormData_pg_attribute *aattr[] = {&attribute}; /* At top level, permission check was done in ATPrepCmd, else do it */ @@ -6422,6 +6422,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) attribute.attcompression = GetAttributeCompression(&attribute, colDef->compression, + &acoptions, NULL); else attribute.attcompression = InvalidOid; @@ -6432,7 +6433,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, tupdesc = CreateTupleDesc(lengthof(aattr), (FormData_pg_attribute **) &aattr); - InsertPgAttributeTuples(attrdesc, tupdesc, myrelid, NULL, NULL); + InsertPgAttributeTuples(attrdesc, tupdesc, myrelid, NULL, &acoptions, NULL); table_close(attrdesc, RowExclusiveLock); @@ -7829,13 +7830,20 @@ ATExecSetOptions(Relation rel, const char *colName, Node *options, * index attribute otherwise it will update the attstorage. */ static void -ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum, - Oid newcompression, char newstorage, LOCKMODE lockmode) +ApplyChangesToIndexes(Relation rel, Relation attrel, AttrNumber attnum, + Oid newcompression, char newstorage, Datum acoptions, + LOCKMODE lockmode) { HeapTuple tuple; ListCell *lc; Form_pg_attribute attrtuple; + /* + * Compression option can only be valid if we are updating the compression + * method. + */ + Assert(DatumGetPointer(acoptions) == NULL || OidIsValid(newcompression)); + foreach(lc, RelationGetIndexList(rel)) { Oid indexoid = lfirst_oid(lc); @@ -7874,7 +7882,29 @@ ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum, else attrtuple->attstorage = newstorage; - CatalogTupleUpdate(attrelation, &tuple->t_self, tuple); + if (DatumGetPointer(acoptions) != NULL) + { + Datum values[Natts_pg_attribute]; + bool nulls[Natts_pg_attribute]; + bool replace[Natts_pg_attribute]; + HeapTuple newtuple; + + /* Initialize buffers for new tuple values */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replace, false, sizeof(replace)); + + values[Anum_pg_attribute_attcmoptions - 1] = acoptions; + replace[Anum_pg_attribute_attcmoptions - 1] = true; + + newtuple = heap_modify_tuple(tuple, RelationGetDescr(attrel), + values, nulls, replace); + CatalogTupleUpdate(attrel, &newtuple->t_self, newtuple); + + heap_freetuple(newtuple); + } + else + CatalogTupleUpdate(attrel, &tuple->t_self, tuple); InvokeObjectPostAlterHook(RelationRelationId, RelationGetRelid(rel), @@ -7965,7 +7995,7 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc * matching behavior of index.c ConstructTupleDescriptor()). */ ApplyChangesToIndexes(rel, attrelation, attnum, InvalidOid, newstorage, - lockmode); + PointerGetDatum(NULL), lockmode); table_close(attrelation, RowExclusiveLock); @@ -15137,9 +15167,11 @@ ATExecSetCompression(AlteredTableInfo *tab, char typstorage; Oid cmoid; bool need_rewrite; + HeapTuple newtuple; Datum values[Natts_pg_attribute]; bool nulls[Natts_pg_attribute]; bool replace[Natts_pg_attribute]; + Datum acoptions; ObjectAddress address; attrel = table_open(AttributeRelationId, RowExclusiveLock); @@ -15174,7 +15206,8 @@ ATExecSetCompression(AlteredTableInfo *tab, memset(replace, false, sizeof(replace)); /* Get the attribute compression method. */ - cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite); + cmoid = GetAttributeCompression(atttableform, compression, &acoptions, + &need_rewrite); if (atttableform->attcompression != cmoid) add_column_compression_dependency(atttableform->attrelid, @@ -15184,8 +15217,17 @@ ATExecSetCompression(AlteredTableInfo *tab, atttableform->attcompression = cmoid; - atttableform->attcompression = cmoid; - CatalogTupleUpdate(attrel, &tuple->t_self, tuple); + /* update an existing entry */ + if (acoptions) + { + values[Anum_pg_attribute_attcmoptions - 1] = acoptions; + replace[Anum_pg_attribute_attcmoptions - 1] = true; + newtuple = heap_modify_tuple(tuple, RelationGetDescr(attrel), + values, nulls, replace); + CatalogTupleUpdate(attrel, &newtuple->t_self, newtuple); + } + else + CatalogTupleUpdate(attrel, &tuple->t_self, tuple); InvokeObjectPostAlterHook(RelationRelationId, RelationGetRelid(rel), @@ -15193,8 +15235,10 @@ ATExecSetCompression(AlteredTableInfo *tab, ReleaseSysCache(tuple); - /* apply changes to the index column as well */ - ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', lockmode); + /* Apply the change to indexes as well */ + ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', acoptions, + lockmode); + table_close(attrel, RowExclusiveLock); /* make changes visible */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 6588c84c4c..69e370de64 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2959,6 +2959,7 @@ _copyColumnCompression(const ColumnCompression *from) COPY_STRING_FIELD(cmname); COPY_SCALAR_FIELD(preserve_all); + COPY_NODE_FIELD(options); COPY_NODE_FIELD(preserve); return newnode; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 3aa6176a30..1e3a01fbc3 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2623,6 +2623,7 @@ _equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b) { COMPARE_STRING_FIELD(cmname); COMPARE_SCALAR_FIELD(preserve_all); + COMPARE_NODE_FIELD(options); COMPARE_NODE_FIELD(preserve); return true; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 4f7053d63b..283452e454 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2890,6 +2890,7 @@ _outColumnCompression(StringInfo str, const ColumnCompression *node) WRITE_STRING_FIELD(cmname); WRITE_BOOL_FIELD(preserve_all); + WRITE_NODE_FIELD(options); WRITE_NODE_FIELD(preserve); } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 0a0ea0562a..2d7b307661 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -415,6 +415,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); relation_expr_list dostmt_opt_list transform_element_list transform_type_list TriggerTransitions TriggerReferencing + optCompressionParameters vacuum_relation_list opt_vacuum_relation_list drop_option_list @@ -3502,11 +3503,17 @@ compressionClause: COMPRESSION name { $$ = pstrdup($2); } ; +optCompressionParameters: + WITH '(' generic_option_list ')' { $$ = $3; } + | /*EMPTY*/ { $$ = NULL; } + ; + optColumnCompression: - compressionClause + compressionClause optCompressionParameters { ColumnCompression *n = makeNode(ColumnCompression); n->cmname = $1; + n->options = (List *) $2; n->preserve = NIL; $$ = (Node *) n; } @@ -3514,14 +3521,15 @@ optColumnCompression: ; alterColumnCompression: - compressionClause optCompressionPreserve + compressionClause optCompressionParameters optCompressionPreserve { ColumnCompression *n = makeNode(ColumnCompression); n->cmname = $1; - n->preserve = (List *) $2; + n->options = (List *) $2; + n->preserve = (List *) $3; $$ = (Node *) n; } - | compressionClause PRESERVE ALL + | compressionClause optCompressionParameters PRESERVE ALL { ColumnCompression *n = makeNode(ColumnCompression); n->cmname = $1; diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index a2a6b218f3..47fddc2ad1 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla /* Likewise, copy compression if requested */ if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION && OidIsValid(attribute->attcompression)) - def->compression = MakeColumnCompression(attribute->attcompression); + def->compression = MakeColumnCompression(attribute); else def->compression = NULL; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3a79754550..7ee41ca57d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -8709,10 +8709,17 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) if (createWithCompression) appendPQExpBuffer(q, - "am.amname AS attcmname,\n"); + "am.amname AS attcmname,\n" + "pg_catalog.array_to_string(ARRAY(" + "SELECT pg_catalog.quote_ident(option_name) || " + "' ' || pg_catalog.quote_literal(option_value) " + "FROM pg_catalog.pg_options_to_table(a.attcmoptions) " + "ORDER BY option_name" + "), E',\n ') AS attcmoptions,\n"); else appendPQExpBuffer(q, - "NULL AS attcmname,\n"); + "NULL AS attcmname,\n" + "NULL AS attcmoptions,\n"); if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, @@ -8765,6 +8772,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8794,6 +8802,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); + tbinfo->attcmoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmoptions"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15932,7 +15941,8 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) continue; has_non_default_compression = (tbinfo->attcmnames[j] && - (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + ((strcmp(tbinfo->attcmnames[j], "pglz") != 0) || + nonemptyReloptions(tbinfo->attcmoptions[j]))); /* Format properly if not first attr */ if (actual_atts == 0) @@ -15983,6 +15993,10 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) { appendPQExpBuffer(q, " COMPRESSION %s", tbinfo->attcmnames[j]); + + if (nonemptyReloptions(tbinfo->attcmoptions[j])) + appendPQExpBuffer(q, " WITH (%s)", + tbinfo->attcmoptions[j]); } if (print_default) @@ -16414,6 +16428,9 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s", qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]); + if (nonemptyReloptions(tbinfo->attcmoptions[j])) + appendPQExpBuffer(q, "\nWITH (%s) ", tbinfo->attcmoptions[j]); + if (cminfo->nitems > 0) { appendPQExpBuffer(q, "\nPRESERVE ("); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 2a751321f5..47ed1112e8 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -328,6 +328,8 @@ typedef struct _tableInfo char *amname; /* relation access method */ char **attcmnames; /* per-attribute current compression method */ + char **attcmoptions; /* per-attribute current compression + options */ struct _attrCompressionInfo **attcompression; /* per-attribute all compression data */ /* * Stuff computed only for dumpable tables. diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h index 36c9f57628..92aba67407 100644 --- a/src/include/access/compressamapi.h +++ b/src/include/access/compressamapi.h @@ -19,8 +19,8 @@ #include "catalog/pg_am_d.h" #include "miscadmin.h" #include "nodes/nodes.h" -#include "utils/guc.h" #include "nodes/pg_list.h" +#include "utils/guc.h" /* * Built-in compression method-id. The toast compression header will store @@ -44,8 +44,11 @@ extern char *default_toast_compression; #define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ (storage) != TYPSTORAGE_EXTERNAL) /* compression handler routines */ +typedef void (*cmcheck_function) (List *options); +typedef void *(*cminitstate_function) (List *options); typedef struct varlena *(*cmcompress_function) (const struct varlena *value, - int32 toast_header_size); + int32 toast_header_size, + void *options); typedef struct varlena *(*cmdecompress_function) (const struct varlena *value, int32 toast_header_size); typedef struct varlena *(*cmdecompress_slice_function) @@ -56,14 +59,25 @@ typedef struct varlena *(*cmdecompress_slice_function) /* * API struct for a compression AM. * + * 'cmcheck' - called when attribute is linking with compression method. + * This function should check compability of compression method with + * the attribute and its options. + * + * 'cminitstate' - called when CompressionAmOptions instance is created. + * Should return pointer to a memory in a caller memory context, or NULL. + * Could be used to pass some internal state between compression function + * calls, like internal structure for parsed compression options. + * * 'datum_compress' - varlena compression function. * 'datum_decompress' - varlena decompression function. * 'datum_decompress_slice' - varlena slice decompression functions. */ typedef struct CompressionAmRoutine { - NodeTag type; + NodeTag type; + cmcheck_function datum_check; /* can be NULL */ + cminitstate_function datum_initstate; /* can be NULL */ cmcompress_function datum_compress; cmdecompress_function datum_decompress; cmdecompress_slice_function datum_decompress_slice; diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index dca0bc37f3..fe8de405ac 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -15,6 +15,7 @@ #define TOAST_HELPER_H #include "utils/rel.h" +#include "nodes/pg_list.h" /* * Information about one column of a tuple being toasted. @@ -33,6 +34,7 @@ typedef struct int32 tai_size; uint8 tai_colflags; Oid tai_compression; + List *tai_cmoptions; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index c158354272..3b143b8c0b 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -64,7 +64,7 @@ typedef struct toast_compress_header_custom #define TOAST_COMPRESS_SET_CMOID(ptr, oid) \ (((toast_compress_header_custom *)(ptr))->cmoid = (oid)) -extern Datum toast_compress_datum(Datum value, Oid cmoid); +extern Datum toast_compress_datum(Datum value, Oid cmoid, List *cmoptions); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index 6ce480b49c..5ecb6f3653 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -81,6 +81,7 @@ extern Oid heap_create_with_catalog(const char *relname, bool allow_system_table_mods, bool is_internal, Oid relrewrite, + Datum *acoptions, ObjectAddress *typaddress); extern void heap_drop_with_catalog(Oid relid); @@ -97,6 +98,7 @@ extern void InsertPgAttributeTuples(Relation pg_attribute_rel, TupleDesc tupdesc, Oid new_rel_oid, Datum *attoptions, + Datum *attcmoptions, CatalogIndexState indstate); extern void InsertPgClassTuple(Relation pg_class_desc, diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index a6cb1b4e5b..71cbc99116 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -176,6 +176,9 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* Column-level FDW options */ text attfdwoptions[1] BKI_DEFAULT(_null_); + /* current compression options */ + text attcmoptions[1] BKI_DEFAULT(_null_); + /* * Missing value for added columns. This is a one element array which lets * us store a value of the attribute type here. diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index bd53f9bb0f..8271e97dea 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -134,6 +134,8 @@ extern Datum transformGenericOptions(Oid catalogId, Datum oldOptions, List *options, Oid fdwvalidator); +extern Datum optionListToArray(List *options); +extern char *formatRelOptions(List *options); /* commands/amcmds.c */ extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); @@ -146,9 +148,12 @@ extern char *get_am_name(Oid amOid); /* commands/compressioncmds.c */ extern Oid GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression, - bool *need_rewrite); -extern ColumnCompression *MakeColumnCompression(Oid atttcompression); + Datum *acoptions, bool *need_rewrite); +extern ColumnCompression *MakeColumnCompression(Form_pg_attribute att); extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid); +extern List *GetAttributeCompressionOptions(Form_pg_attribute att); +extern void CheckCompressionMismatch(ColumnCompression *c1, ColumnCompression *c2, + const char *attributeName); /* support routines in commands/define.c */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index fd9c6abddf..a1b1158316 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -634,6 +634,7 @@ typedef struct ColumnCompression NodeTag type; char *cmname; bool preserve_all; + List *options; List *preserve; } ColumnCompression; diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 636ab4021b..9d2c00fb73 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -272,6 +272,58 @@ SELECT pg_column_compression(f1) FROM cmdata; Indexes: "idx" btree (f1) +-- compression options +CREATE TABLE cmdata3(f1 TEXT COMPRESSION pglz WITH (min_input_size '100')); +CREATE INDEX idx1 ON cmdata3(f1); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1000)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION lz4 WITH (acceleration '50'); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1004)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; + attcmoptions +------------------- + {acceleration=50} +(1 row) + +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + attcmoptions +------------------- + {acceleration=50} +(1 row) + +ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION pglz WITH (min_input_size '200') PRESERVE (lz4); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1008)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=200} +(1 row) + +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=200} +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata3; + pg_column_compression +----------------------- + lz4 + lz4 + pglz +(3 rows) + -- check data is ok SELECT length(f1) FROM cmdata; length diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index 3177ff91e8..4c0952bc0a 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -270,6 +270,61 @@ SELECT pg_column_compression(f1) FROM cmdata; Indexes: "idx" btree (f1) +-- compression options +CREATE TABLE cmdata3(f1 TEXT COMPRESSION pglz WITH (min_input_size '100')); +CREATE INDEX idx1 ON cmdata3(f1); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1000)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION lz4 WITH (acceleration '50'); +ERROR: not built with lz4 support +INSERT INTO cmdata3 VALUES(repeat('1234567890',1004)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION pglz WITH (min_input_size '200') PRESERVE (lz4); +ERROR: "lz4" compression access method cannot be preserved +HINT: use "pg_column_compression" function for list of compression methods +INSERT INTO cmdata3 VALUES(repeat('1234567890',1008)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + attcmoptions +---------------------- + {min_input_size=100} +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata3; + pg_column_compression +----------------------- + pglz + pglz + pglz +(3 rows) + -- check data is ok SELECT length(f1) FROM cmdata; length diff --git a/src/test/regress/expected/misc_sanity.out b/src/test/regress/expected/misc_sanity.out index d40afeef78..27aab82462 100644 --- a/src/test/regress/expected/misc_sanity.out +++ b/src/test/regress/expected/misc_sanity.out @@ -97,6 +97,7 @@ ORDER BY 1, 2; relname | attname | atttypid -------------------------+---------------+-------------- pg_attribute | attacl | aclitem[] + pg_attribute | attcmoptions | text[] pg_attribute | attfdwoptions | text[] pg_attribute | attmissingval | anyarray pg_attribute | attoptions | text[] @@ -107,5 +108,5 @@ ORDER BY 1, 2; pg_index | indpred | pg_node_tree pg_largeobject | data | bytea pg_largeobject_metadata | lomacl | aclitem[] -(11 rows) +(12 rows) diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 0552eeca16..59a32dd0a3 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -105,6 +105,24 @@ ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2); SELECT pg_column_compression(f1) FROM cmdata; \d+ cmdata +-- compression options +CREATE TABLE cmdata3(f1 TEXT COMPRESSION pglz WITH (min_input_size '100')); +CREATE INDEX idx1 ON cmdata3(f1); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1000)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + +ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION lz4 WITH (acceleration '50'); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1004)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; + +ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION pglz WITH (min_input_size '200') PRESERVE (lz4); +INSERT INTO cmdata3 VALUES(repeat('1234567890',1008)); +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1'; +SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1'; +SELECT pg_column_compression(f1) FROM cmdata3; + -- check data is ok SELECT length(f1) FROM cmdata; SELECT length(f1) FROM cmdata1; -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-Add-default_toast_compression-GUC.beforerebase" From 4b5a3855e1900a79d1b65d3606c7e9f165605d51 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 29 Jan 2021 21:37:46 -0600 Subject: [PATCH] Add default_toast_compression GUC --- src/backend/access/common/tupdesc.c | 3 +- .../access/compression/compressamapi.c | 54 +++++++++++++++++++ src/backend/bootstrap/bootstrap.c | 7 ++- src/backend/commands/compressioncmds.c | 2 +- src/backend/commands/tablecmds.c | 2 +- src/backend/utils/cache/syscache.c | 6 ++- src/backend/utils/misc/guc.c | 12 +++++ src/include/access/compressamapi.h | 12 ++++- 8 files changed, 91 insertions(+), 7 deletions(-) diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index ca26fab487..b56b689493 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -25,6 +25,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" +#include "commands/defrem.h" #include "miscadmin.h" #include "parser/parse_type.h" #include "utils/acl.h" @@ -668,7 +669,7 @@ TupleDescInitEntry(TupleDesc desc, att->attcollation = typeForm->typcollation; if (IsStorageCompressible(typeForm->typstorage)) - att->attcompression = DefaultCompressionOid; + att->attcompression = get_compression_am_oid(default_toast_compression, false); else att->attcompression = InvalidOid; diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c index 7aea8aad38..19780f57e7 100644 --- a/src/backend/access/compression/compressamapi.c +++ b/src/backend/access/compression/compressamapi.c @@ -20,9 +20,12 @@ #include "access/reloptions.h" #include "access/table.h" #include "catalog/pg_am.h" +#include "commands/defrem.h" #include "utils/fmgroids.h" #include "utils/syscache.h" +char *default_toast_compression = DEFAULT_TOAST_COMPRESSION; // maybe needs pg_dump support ? + /* * CompressionOidToId - Convert compression Oid to built-in compression id. * @@ -86,3 +89,54 @@ GetCompressionAmRoutineByAmId(Oid amoid) return routine; } + +/* check_hook: validate new default_toast_compression */ +bool +check_default_toast_compression(char **newval, void **extra, GucSource source) +{ + if (**newval == '\0') + { + GUC_check_errdetail("%s cannot be empty.", + "default_toast_compression"); + return false; + } + + if (strlen(*newval) >= NAMEDATALEN) + { + GUC_check_errdetail("%s is too long (maximum %d characters).", + "default_toast_compression", NAMEDATALEN - 1); + return false; + } + + /* + * If we aren't inside a transaction, or not connected to a database, we + * cannot do the catalog access necessary to verify the method. Must + * accept the value on faith. + */ + if (IsTransactionState() && MyDatabaseId != InvalidOid) + { + if (!OidIsValid(get_compression_am_oid(*newval, true))) + { + /* + * When source == PGC_S_TEST, don't throw a hard error for a + * nonexistent table access method, only a NOTICE. See comments in + * guc.h. + */ + if (source == PGC_S_TEST) + { + ereport(NOTICE, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("compression access method \"%s\" does not exist", + *newval))); + } + else + { + GUC_check_errdetail("Compression access method \"%s\" does not exist.", + *newval); + return false; + } + } + } + + return true; +} diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 9b451eaa71..1b885f873a 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -29,6 +29,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/link-canary.h" +#include "commands/defrem.h" #include "libpq/pqsignal.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -732,8 +733,12 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) - attrtypes[attnum]->attcompression = DefaultCompressionOid; + { + /* Cannot call get_compression_am_oid this early */ + attrtypes[attnum]->attcompression = PGLZ_COMPRESSION_AM_OID; + } else attrtypes[attnum]->attcompression = InvalidOid; diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c index de57b55aab..d70408c7c2 100644 --- a/src/backend/commands/compressioncmds.c +++ b/src/backend/commands/compressioncmds.c @@ -204,7 +204,7 @@ GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression, /* fallback to default compression if it's not specified */ if (compression == NULL) - return DefaultCompressionOid; + return get_compression_am_oid(default_toast_compression, false); cmoid = get_compression_am_oid(compression->cmname, false); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index cc3a0cb753..93c1374bd4 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -12035,7 +12035,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, if (!IsStorageCompressible(tform->typstorage)) attTup->attcompression = InvalidOid; else if (!OidIsValid(attTup->attcompression)) - attTup->attcompression = DefaultCompressionOid; + attTup->attcompression = get_compression_am_oid(default_toast_compression, false); } else attTup->attcompression = InvalidOid; diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c index e4dc4ee34e..74e42a3394 100644 --- a/src/backend/utils/cache/syscache.c +++ b/src/backend/utils/cache/syscache.c @@ -1127,8 +1127,10 @@ HeapTuple SearchSysCache1(int cacheId, Datum key1) { - Assert(cacheId >= 0 && cacheId < SysCacheSize && - PointerIsValid(SysCache[cacheId])); + Assert(cacheId >= 0); + Assert(cacheId < SysCacheSize); + // fprintf(stderr, "cache %d\n", cacheId); + Assert(PointerIsValid(SysCache[cacheId])); Assert(SysCache[cacheId]->cc_nkeys == 1); return SearchCatCache1(SysCache[cacheId], key1); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index eafdb1118e..70b26ea72a 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -30,6 +30,7 @@ #include #include "access/commit_ts.h" +#include "access/compressamapi.h" #include "access/gin.h" #include "access/rmgr.h" #include "access/tableam.h" @@ -3915,6 +3916,17 @@ static struct config_string ConfigureNamesString[] = check_default_table_access_method, NULL, NULL }, + { + {"default_toast_compression", PGC_USERSET, CLIENT_CONN_STATEMENT, + gettext_noop("Sets the default compression for new columns."), + NULL, + GUC_IS_NAME + }, + &default_toast_compression, + DEFAULT_TOAST_COMPRESSION, + check_default_toast_compression, NULL, NULL, NULL + }, + { {"default_tablespace", PGC_USERSET, CLIENT_CONN_STATEMENT, gettext_noop("Sets the default tablespace to create tables and indexes in."), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h index 8123cc8cc7..92aba67407 100644 --- a/src/include/access/compressamapi.h +++ b/src/include/access/compressamapi.h @@ -15,9 +15,12 @@ #include "postgres.h" +#include "access/xact.h" #include "catalog/pg_am_d.h" +#include "miscadmin.h" #include "nodes/nodes.h" #include "nodes/pg_list.h" +#include "utils/guc.h" /* * Built-in compression method-id. The toast compression header will store @@ -32,8 +35,11 @@ typedef enum CompressionId CUSTOM_COMPRESSION_ID = 3 } CompressionId; +#define DEFAULT_TOAST_COMPRESSION "pglz" + +extern char *default_toast_compression; + /* Use default compression method if it is not specified. */ -#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID #define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID) #define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ (storage) != TYPSTORAGE_EXTERNAL) @@ -85,4 +91,8 @@ extern CompressionId CompressionOidToId(Oid cmoid); extern Oid CompressionIdToOid(CompressionId cmid); extern CompressionAmRoutine *GetCompressionAmRoutineByAmId(Oid amoid); +extern void assign_default_toast_compression(const char *newval, void **extra); +extern bool check_default_toast_compression(char **newval, void **extra, GucSource source); + + #endif /* COMPRESSAMAPI_H */ -- 2.17.0 --ytoMbUMiTKPMT3hY--