($INBOX_DIR/description missing)
help / color / mirror / Atom feed[PATCH v21 4/7] Add support for PRESERVE
84+ messages / 12 participants
[nested] [flat]
* [PATCH v21 4/7] Add support for PRESERVE
@ 2021-01-04 09:45 dilipkumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: dilipkumar @ 2021-01-04 09:45 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
Discussions:
https://www.postgresql.org/message-id/[email protected]
https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40...
https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail...
https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development
https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail...
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 293 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 116 ++++----
src/backend/executor/nodeModifyTable.c | 13 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 14 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 732 insertions(+), 98 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c4b53ec1c5..081fe078a4 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -386,13 +386,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
This clause adds compression to a column. The Compression method can be
set to any available compression method. The built-in methods are
<literal>pglz</literal> and <literal>lz4</literal>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..50f0f7faaa
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,293 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ return InvalidOid;
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return get_compression_am_oid(default_toast_compression, false);
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p),
+ errhint("use \"pg_column_compression\" function for list of compression methods")));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 89cca760c5..9d732fc167 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression))
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6402,7 +6421,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6577,6 +6597,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6724,6 +6745,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11859,7 +11902,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11974,6 +12018,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15078,24 +15127,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15128,11 +15174,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17857,32 +17908,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * Get compression method Oid for the attribute.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression != NULL)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- return InvalidOid;
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return get_compression_am_oid(default_toast_compression, false);
-
- return get_compression_am_oid(compression, false);
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 684e836071..916f6541c1 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -45,6 +45,7 @@
#include "access/toast_internals.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2054,6 +2055,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
int natts = slot->tts_tupleDescriptor->natts;
bool isnull = false;
bool decompressed_any = false;
+ Oid cmoid = InvalidOid;
TupleDesc tupleDesc = slot->tts_tupleDescriptor;
if (natts == 0)
@@ -2062,7 +2064,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop for all the attributes in the tuple and check if any of the
* attribute is compressed in the source tuple and its compression method
- * is not same as the target compression method then we need to decompress
+ * is not supported by the target attribute then we need to decompress
* it.
*/
for (i = 0; i < natts; i++)
@@ -2083,11 +2085,12 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
continue;
/*
- * Get the compression method stored in the toast header and
- * compare with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
- if (targetTupDesc->attrs[i].attcompression !=
- CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value)))
+ cmoid = CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value));
+ if (!IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 3ea32323cd..6588c84c4c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2932,7 +2932,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2952,6 +2952,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5640,6 +5652,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 227bb07bbd..3aa6176a30 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2598,7 +2598,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2618,6 +2618,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3693,6 +3703,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 187d3570e5..4f7053d63b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2865,7 +2865,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2883,6 +2883,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4230,6 +4240,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index f2e9fe4ff8..332e60f170 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -595,7 +595,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2308,12 +2310,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);;
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3436,7 +3438,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3491,13 +3493,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index e2ac159aa2..a2a6b218f3 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index becf565cdd..c09f3972c6 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9010,6 +9010,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16326,6 +16400,33 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 53ece7abd7..2a751321f5 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -328,7 +328,7 @@ typedef struct _tableInfo
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -357,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 7aeabf9543..c40d498a30 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index ded2ad9692..d630c54efa 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -479,6 +479,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ffb739f565..fd9c6abddf 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 84d31c781b..151478bcdd 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -201,12 +201,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 3250c12859..81182fe949 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -196,12 +196,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
pglz
(1 row)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+HINT: use "pg_column_compression" function for list of compression methods
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index ce734f7ef3..b61c6278a2 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 3b0da88c19..e447316573 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -86,6 +86,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--ytoMbUMiTKPMT3hY
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v21-0005-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH 08/12] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 5e24af9cb3..5fba7352a3 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index e262ec5bcb..f50b50cc8e 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2aedddc1d8..05c782e8e9 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2112,6 +2112,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--9Ek0hoCL9XbhcSqy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0009-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 7/7] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--YZ5djTAD1cGYuMQK--
^ permalink raw reply [nested|flat] 84+ messages in thread
* [PATCH v24 07/10] Add support for PRESERVE
@ 2021-02-10 06:19 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Dilip Kumar @ 2021-02-10 06:19 UTC (permalink / raw)
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. �For
supporting this the column will maintain the dependency with all
the supported compression methods. �So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. �If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 586a92f0c1..2a1841c353 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index cf4413da64..45f4724a13 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ffa8d05edf..869fd3676d 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2103,6 +2103,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0008-Create-custom-compression-methods.patch"
^ permalink raw reply [nested|flat] 84+ messages in thread
* sequences vs. synchronous replication
@ 2021-12-18 01:53 Tomas Vondra <[email protected]>
0 siblings, 2 replies; 84+ messages in thread
From: Tomas Vondra @ 2021-12-18 01:53 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Hi,
while working on logical decoding of sequences, I ran into an issue with
nextval() in a transaction that rolls back, described in [1]. But after
thinking about it a bit more (and chatting with Petr Jelinek), I think
this issue affects physical sync replication too.
Imagine you have a primary <-> sync_replica cluster, and you do this:
CREATE SEQUENCE s;
-- shutdown the sync replica
BEGIN;
SELECT nextval('s') FROM generate_series(1,50);
ROLLBACK;
BEGIN;
SELECT nextval('s');
COMMIT;
The natural expectation would be the COMMIT gets stuck, waiting for the
sync replica (which is not running), right? But it does not.
The problem is exactly the same as in [1] - the aborted transaction
generated WAL, but RecordTransactionAbort() ignores that and does not
update LogwrtResult.Write, with the reasoning that aborted transactions
do not matter. But sequences violate that, because we only write WAL
once every 32 increments, so the following nextval() gets "committed"
without waiting for the replica (because it did not produce WAL).
I'm not sure this is a clear data corruption bug, but it surely walks
and quacks like one. My proposal is to fix this by tracking the lsn of
the last LSN for a sequence increment, and then check that LSN in
RecordTransactionCommit() before calling XLogFlush().
regards
[1]
https://www.postgresql.org/message-id/ae3cab67-c31e-b527-dd73-08f196999ad4%40enterprisedb.com
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-18 04:52 Tom Lane <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 84+ messages in thread
From: Tom Lane @ 2021-12-18 04:52 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Tomas Vondra <[email protected]> writes:
> The problem is exactly the same as in [1] - the aborted transaction
> generated WAL, but RecordTransactionAbort() ignores that and does not
> update LogwrtResult.Write, with the reasoning that aborted transactions
> do not matter. But sequences violate that, because we only write WAL
> once every 32 increments, so the following nextval() gets "committed"
> without waiting for the replica (because it did not produce WAL).
Ugh.
> I'm not sure this is a clear data corruption bug, but it surely walks
> and quacks like one. My proposal is to fix this by tracking the lsn of
> the last LSN for a sequence increment, and then check that LSN in
> RecordTransactionCommit() before calling XLogFlush().
(1) Does that work if the aborted increment was in a different
session? I think it is okay but I'm tired enough to not be sure.
(2) I'm starting to wonder if we should rethink the sequence logging
mechanism altogether. It was cool when designed, but it seems
really problematic when you start thinking about replication
behaviors. Perhaps if wal_level > minimal, we don't do things
the same way?
regards, tom lane
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-18 06:00 Tomas Vondra <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-18 06:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/18/21 05:52, Tom Lane wrote:
> Tomas Vondra <[email protected]> writes:
>> The problem is exactly the same as in [1] - the aborted transaction
>> generated WAL, but RecordTransactionAbort() ignores that and does not
>> update LogwrtResult.Write, with the reasoning that aborted transactions
>> do not matter. But sequences violate that, because we only write WAL
>> once every 32 increments, so the following nextval() gets "committed"
>> without waiting for the replica (because it did not produce WAL).
>
> Ugh.
>
>> I'm not sure this is a clear data corruption bug, but it surely walks
>> and quacks like one. My proposal is to fix this by tracking the lsn of
>> the last LSN for a sequence increment, and then check that LSN in
>> RecordTransactionCommit() before calling XLogFlush().
>
> (1) Does that work if the aborted increment was in a different
> session? I think it is okay but I'm tired enough to not be sure.
>
Good point - it doesn't :-( At least not by simply storing LSN in a
global variable or something like that.
The second backend needs to know the LSN of the last WAL-logged sequence
increment, but only the first backend knows that. So we'd need to share
that between backends somehow. I doubt we want to track LSN for every
individual sequence (because for clusters with many dbs / sequences that
may be a lot).
Perhaps we could track just a fixed number o LSN values in shared memory
(say, 1024), and update/read just the element determined by hash(oid).
That is, the backend WAL-logging sequence with given oid would set the
current LSN to array[hash(oid) % 1024], and backend doing nextval()
would simply remember the LSN in that slot. Yes, if there are conflicts
that'll flush more than needed.
Alternatively we could simply use the current insert LSN, but that's
going to flush more stuff than needed all the time.
> (2) I'm starting to wonder if we should rethink the sequence logging
> mechanism altogether. It was cool when designed, but it seems
> really problematic when you start thinking about replication
> behaviors. Perhaps if wal_level > minimal, we don't do things
> the same way?
Maybe, but I have no idea how should the reworked WAL logging work. Any
batching seems to have this issue, and loging individual increments is
likely going to be slower.
Of course, reworking how sequences are WAL-logged may invalidate the
"sequence decoding" patch I've been working on :-(
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-18 20:45 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-18 20:45 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/18/21 07:00, Tomas Vondra wrote:
>
>
> On 12/18/21 05:52, Tom Lane wrote:
>> Tomas Vondra <[email protected]> writes:
>>> The problem is exactly the same as in [1] - the aborted transaction
>>> generated WAL, but RecordTransactionAbort() ignores that and does not
>>> update LogwrtResult.Write, with the reasoning that aborted transactions
>>> do not matter. But sequences violate that, because we only write WAL
>>> once every 32 increments, so the following nextval() gets "committed"
>>> without waiting for the replica (because it did not produce WAL).
>>
>> Ugh.
>>
>>> I'm not sure this is a clear data corruption bug, but it surely walks
>>> and quacks like one. My proposal is to fix this by tracking the lsn of
>>> the last LSN for a sequence increment, and then check that LSN in
>>> RecordTransactionCommit() before calling XLogFlush().
>>
>> (1) Does that work if the aborted increment was in a different
>> session? I think it is okay but I'm tired enough to not be sure.
>>
>
> Good point - it doesn't :-( At least not by simply storing LSN in a
> global variable or something like that.
>
> The second backend needs to know the LSN of the last WAL-logged sequence
> increment, but only the first backend knows that. So we'd need to share
> that between backends somehow. I doubt we want to track LSN for every
> individual sequence (because for clusters with many dbs / sequences that
> may be a lot).
>
> Perhaps we could track just a fixed number o LSN values in shared memory
> (say, 1024), and update/read just the element determined by hash(oid).
> That is, the backend WAL-logging sequence with given oid would set the
> current LSN to array[hash(oid) % 1024], and backend doing nextval()
> would simply remember the LSN in that slot. Yes, if there are conflicts
> that'll flush more than needed.
>
Here's a PoC demonstrating this idea. I'm not convinced it's the right
way to deal with this - it surely seems more like a duct tape fix than a
clean solution. But it does the trick.
I wonder if storing this in shmem is good enough - we lose the LSN info
on restart, but the checkpoint should trigger FPI which makes it OK.
A bigger question is whether sequences are the only thing affected by
this. If you look at RecordTransactionCommit() then we skip flush/wait
in two cases:
1) !wrote_xlog - if the xact did not produce WAL
2) !markXidCommitted - if the xact does not have a valid XID
Both apply to sequences, and the PoC patch tweaks them. But maybe there
are other places where we don't generate WAL and/or assign XID in some
cases, to save time?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[text/x-patch] sequences-track-lsn-poc.patch (8.3K, ../../[email protected]/2-sequences-track-lsn-poc.patch)
download | inline diff:
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index e7b0bc804d..c6a0c07846 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -35,6 +35,7 @@
#include "catalog/pg_enum.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/sequence.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "common/pg_prng.h"
@@ -1289,6 +1290,13 @@ RecordTransactionCommit(void)
bool RelcacheInitFileInval = false;
bool wrote_xlog;
+ /*
+ * Force flush and synchronous replication even if no XID was assigned.
+ * This is needed when depending on WAL produced by another transaction
+ * (possibly in a different backend). Sequences need this, for example.
+ */
+ bool force_sync = false;
+
/*
* Log pending invalidations for logical decoding of in-progress
* transactions. Normally for DDLs, we log this at each command end,
@@ -1299,6 +1307,24 @@ RecordTransactionCommit(void)
if (XLogLogicalInfoActive())
LogLogicalInvalidations();
+ /*
+ * Check the LSN of increments for sequences we touched in this transaction.
+ * If it's higher than XactLastRecEnd, we need to force flush/sync.
+ */
+ {
+ /* Separate call, so that we can compare to XactLastRecEnd. */
+ XLogRecPtr tmpptr = SequenceGetLastLSN();
+
+ /*
+ * If higher than XactLastRecEnd, make sure we flush even without
+ * a XID assigned.
+ */
+ force_sync = (tmpptr > XactLastRecEnd);
+
+ /* Override the XactLastRecEnd value. */
+ XactLastRecEnd = Max(XactLastRecEnd, tmpptr);
+ }
+
/* Get data needed for commit record */
nrels = smgrGetPendingDeletes(true, &rels);
nchildren = xactGetCommittedChildren(&children);
@@ -1446,7 +1472,7 @@ RecordTransactionCommit(void)
* if all to-be-deleted tables are temporary though, since they are lost
* anyway if we crash.)
*/
- if ((wrote_xlog && markXidCommitted &&
+ if ((wrote_xlog && (markXidCommitted || force_sync) &&
synchronous_commit > SYNCHRONOUS_COMMIT_OFF) ||
forceSyncCommit || nrels > 0)
{
@@ -1504,7 +1530,7 @@ RecordTransactionCommit(void)
* Note that at this stage we have marked clog, but still show as running
* in the procarray and continue to hold locks.
*/
- if (wrote_xlog && markXidCommitted)
+ if (wrote_xlog && (markXidCommitted || force_sync))
SyncRepWaitForLSN(XactLastRecEnd, true);
/* remember end of last commit record */
@@ -1512,6 +1538,8 @@ RecordTransactionCommit(void)
/* Reset XactLastRecEnd until the next transaction writes something */
XactLastRecEnd = 0;
+ SequenceResetLastLSN();
+
cleanup:
/* Clean up local data */
if (rels)
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index 72bfdc07a4..43fe19831e 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -109,6 +109,19 @@ static void init_params(ParseState *pstate, List *options, bool for_identity,
static void do_setval(Oid relid, int64 next, bool iscalled);
static void process_owned_by(Relation seqrel, List *owned_by, bool for_identity);
+/* handle flush/sync for sequences */
+#define SEQUENCE_LSN_COUNT 1024
+
+typedef struct SequenceLsnState {
+ slock_t lock;
+ XLogRecPtr lsn[SEQUENCE_LSN_COUNT];
+} SequenceLsnState;
+
+static SequenceLsnState *SequenceShmemInfo = NULL;
+static XLogRecPtr lastSequenceLSN = InvalidXLogRecPtr;
+
+static void SequenceSetLSN(Oid relid, XLogRecPtr recptr);
+static XLogRecPtr SequenceGetLSN(Oid relid);
/*
* DefineSequence
@@ -406,7 +419,12 @@ fill_seq_with_data(Relation rel, HeapTuple tuple)
recptr = XLogInsert(RM_SEQ_ID, XLOG_SEQ_LOG);
PageSetLSN(page, recptr);
+
+ SequenceSetLSN(rel->rd_id, recptr);
+ lastSequenceLSN = recptr;
}
+ else
+ lastSequenceLSN = SequenceGetLSN(rel->rd_id);
END_CRIT_SECTION();
@@ -810,7 +828,12 @@ nextval_internal(Oid relid, bool check_permissions)
recptr = XLogInsert(RM_SEQ_ID, XLOG_SEQ_LOG);
PageSetLSN(page, recptr);
+
+ SequenceSetLSN(relid, recptr);
+ lastSequenceLSN = recptr;
}
+ else
+ lastSequenceLSN = SequenceGetLSN(relid);
/* Now update sequence tuple to the intended final state */
seq->last_value = last; /* last fetched number */
@@ -1005,7 +1028,12 @@ do_setval(Oid relid, int64 next, bool iscalled)
recptr = XLogInsert(RM_SEQ_ID, XLOG_SEQ_LOG);
PageSetLSN(page, recptr);
+
+ SequenceSetLSN(relid, recptr);
+ lastSequenceLSN = recptr;
}
+ else
+ lastSequenceLSN = SequenceGetLSN(relid);
END_CRIT_SECTION();
@@ -1927,3 +1955,98 @@ seq_mask(char *page, BlockNumber blkno)
mask_unused_space(page);
}
+
+
+/*
+ * Initialization of shared memory for sequence LSN positions.
+ *
+ * To track the last LSN for each sequence we use a fixed-length array, and use
+ * OID of the sequence to select the element. This allows us to handle unknown
+ * number of sequences in a limited amount of space, but it may force us to
+ * sync a bit too much if two sequences conflict (pick the same array element).
+ * But that seems like acceptable tradeoff.
+ */
+Size
+SequenceShmemSize(void)
+{
+ return sizeof(SequenceLsnState);
+}
+
+/*
+ * Initialize the LSN array (set it to InvalidXLogRecPtr).
+ */
+void
+SequenceShmemInit(void)
+{
+ bool found;
+ SequenceShmemInfo
+ = (SequenceLsnState *) ShmemInitStruct("Sequences Shmem", SequenceShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ SpinLockInit(&(SequenceShmemInfo->lock));
+ MemSet(SequenceShmemInfo->lsn, 0, SEQUENCE_LSN_COUNT * sizeof(XLogRecPtr));
+ }
+}
+
+/*
+ * Update LSN tracked for a sequence with a given OID.
+ *
+ * XXX Maybe hash the OID.
+ */
+static void
+SequenceSetLSN(Oid relid, XLogRecPtr recptr)
+{
+ int idx = (relid % SEQUENCE_LSN_COUNT);
+
+ Assert(SequenceShmemInfo);
+
+ SpinLockAcquire(&(SequenceShmemInfo->lock));
+ SequenceShmemInfo->lsn[idx] = Max(SequenceShmemInfo->lsn[idx], recptr);
+ SpinLockRelease(&(SequenceShmemInfo->lock));
+}
+
+/*
+ * Get LSN tracked for a sequence with a given OID.
+ *
+ * XXX What if the LSN is 0/0? Presumably that means we don't have any value
+ * (e.g. after a restart). Maybe we should use current insert LSN in that case?
+ * Although, shutdown forces checkpoint, and we write FPI or something?
+ */
+static XLogRecPtr
+SequenceGetLSN(Oid relid)
+{
+ int idx = (relid % SEQUENCE_LSN_COUNT);
+ XLogRecPtr lsn;
+
+ Assert(SequenceShmemInfo);
+
+ SpinLockAcquire(&(SequenceShmemInfo->lock));
+ lsn = SequenceShmemInfo->lsn[idx];
+ SpinLockRelease(&(SequenceShmemInfo->lock));
+
+ return lsn;
+}
+
+/*
+ * Return the higher LSN for any sequence we touched in this session (either
+ * the WAL we wrote, or WAL produced by other transaction we depend on).
+ */
+XLogRecPtr
+SequenceGetLastLSN(void)
+{
+ return lastSequenceLSN;
+}
+
+/*
+ * Reset the last LSN after we flush.
+ *
+ * XXX Not sure we actually need to do this. Maybe we don't need tp reset and
+ * keeping the value forever is fine.
+ */
+void
+SequenceResetLastLSN(void)
+{
+ lastSequenceLSN = InvalidXLogRecPtr;
+}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 9fa3e0631e..517cca450b 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -23,6 +23,7 @@
#include "access/syncscan.h"
#include "access/twophase.h"
#include "commands/async.h"
+#include "commands/sequence.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -143,6 +144,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, SequenceShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -293,6 +295,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ SequenceShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/include/commands/sequence.h b/src/include/commands/sequence.h
index 40544dd4c7..85fbc3789d 100644
--- a/src/include/commands/sequence.h
+++ b/src/include/commands/sequence.h
@@ -66,4 +66,10 @@ extern void seq_desc(StringInfo buf, XLogReaderState *rptr);
extern const char *seq_identify(uint8 info);
extern void seq_mask(char *pagedata, BlockNumber blkno);
+
+extern Size SequenceShmemSize(void);
+extern void SequenceShmemInit(void);
+extern XLogRecPtr SequenceGetLastLSN(void);
+extern void SequenceResetLastLSN(void);
+
#endif /* SEQUENCE_H */
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-18 21:27 Tom Lane <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tom Lane @ 2021-12-18 21:27 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Tomas Vondra <[email protected]> writes:
> Here's a PoC demonstrating this idea. I'm not convinced it's the right
> way to deal with this - it surely seems more like a duct tape fix than a
> clean solution. But it does the trick.
I was imagining something a whole lot simpler, like "don't try to
cache unused sequence numbers when wal_level > minimal". We've
accepted worse performance hits in that operating mode, and it'd
fix a number of user complaints we've seen about weird sequence
behavior on standbys.
regards, tom lane
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-18 21:48 Tomas Vondra <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 84+ messages in thread
From: Tomas Vondra @ 2021-12-18 21:48 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/18/21 22:27, Tom Lane wrote:
> Tomas Vondra <[email protected]> writes:
>> Here's a PoC demonstrating this idea. I'm not convinced it's the right
>> way to deal with this - it surely seems more like a duct tape fix than a
>> clean solution. But it does the trick.
>
> I was imagining something a whole lot simpler, like "don't try to
> cache unused sequence numbers when wal_level > minimal". We've
> accepted worse performance hits in that operating mode, and it'd
> fix a number of user complaints we've seen about weird sequence
> behavior on standbys.
>
What do you mean by "not caching unused sequence numbers"? Reducing
SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment?
That'd work, but I wonder how significant the impact will be. It'd bet
it hurts the patch adding logical decoding of sequences quite a bit.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-18 21:51 Tom Lane <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 84+ messages in thread
From: Tom Lane @ 2021-12-18 21:51 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Tomas Vondra <[email protected]> writes:
> What do you mean by "not caching unused sequence numbers"? Reducing
> SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment?
Right.
> That'd work, but I wonder how significant the impact will be.
As I said, we've accepted worse in order to have stable replication
behavior.
regards, tom lane
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-19 03:03 Amit Kapila <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 84+ messages in thread
From: Amit Kapila @ 2021-12-19 03:03 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra
<[email protected]> wrote:
>
> while working on logical decoding of sequences, I ran into an issue with
> nextval() in a transaction that rolls back, described in [1]. But after
> thinking about it a bit more (and chatting with Petr Jelinek), I think
> this issue affects physical sync replication too.
>
> Imagine you have a primary <-> sync_replica cluster, and you do this:
>
> CREATE SEQUENCE s;
>
> -- shutdown the sync replica
>
> BEGIN;
> SELECT nextval('s') FROM generate_series(1,50);
> ROLLBACK;
>
> BEGIN;
> SELECT nextval('s');
> COMMIT;
>
> The natural expectation would be the COMMIT gets stuck, waiting for the
> sync replica (which is not running), right? But it does not.
>
How about if we always WAL log the first sequence change in a transaction?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-20 14:31 Peter Eisentraut <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 84+ messages in thread
From: Peter Eisentraut @ 2021-12-20 14:31 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 18.12.21 22:48, Tomas Vondra wrote:
> What do you mean by "not caching unused sequence numbers"? Reducing
> SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment?
>
> That'd work, but I wonder how significant the impact will be. It'd bet
> it hurts the patch adding logical decoding of sequences quite a bit.
It might be worth testing. This behavior is ancient and has never
really been scrutinized since it was added.
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-20 16:40 Tomas Vondra <[email protected]>
parent: Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-20 16:40 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/20/21 15:31, Peter Eisentraut wrote:
> On 18.12.21 22:48, Tomas Vondra wrote:
>> What do you mean by "not caching unused sequence numbers"? Reducing
>> SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment?
>>
>> That'd work, but I wonder how significant the impact will be. It'd bet
>> it hurts the patch adding logical decoding of sequences quite a bit.
>
> It might be worth testing. This behavior is ancient and has never
> really been scrutinized since it was added.
>
OK, I'll do some testing to measure the overhead, and I'll see how much
it affects the sequence decoding patch.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-21 00:53 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-21 00:53 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/20/21 17:40, Tomas Vondra wrote:
> On 12/20/21 15:31, Peter Eisentraut wrote:
>> On 18.12.21 22:48, Tomas Vondra wrote:
>>> What do you mean by "not caching unused sequence numbers"? Reducing
>>> SEQ_LOG_VALS to 1, i.e. WAL-logging every sequence increment?
>>>
>>> That'd work, but I wonder how significant the impact will be. It'd
>>> bet it hurts the patch adding logical decoding of sequences quite a bit.
>>
>> It might be worth testing. This behavior is ancient and has never
>> really been scrutinized since it was added.
>>
>
> OK, I'll do some testing to measure the overhead, and I'll see how much
> it affects the sequence decoding patch.
>
OK, I did a quick test with two very simple benchmarks - simple select
from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current
master, patched means SEQ_LOG_VALS was set to 1.
Average of 10 runs, each 30 seconds long, look like this:
1) select nextval('s');
clients 1 4
------------------------------
master 39497 123137
patched 6813 18326
------------------------------
diff -83% -86%
2) pgbench -N
clients 1 4
------------------------------
master 2935 9156
patched 2937 9100
------------------------------
diff 0% 0%
Clearly the extreme case (1) is hit pretty bad, while the much mure
likely workload (2) is almost unaffected.
I'm not sure what conclusion to make from this, but assuming almost no
one does just nextval calls, it should be acceptable.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-21 01:01 Tom Lane <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tom Lane @ 2021-12-21 01:01 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Tomas Vondra <[email protected]> writes:
> OK, I did a quick test with two very simple benchmarks - simple select
> from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current
> master, patched means SEQ_LOG_VALS was set to 1.
But ... pgbench -N doesn't use sequences at all, does it?
Probably inserts into a table with a serial column would constitute a
plausible real-world case.
regards, tom lane
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-21 02:49 Tomas Vondra <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-21 02:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 12/21/21 02:01, Tom Lane wrote:
> Tomas Vondra <[email protected]> writes:
>> OK, I did a quick test with two very simple benchmarks - simple select
>> from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current
>> master, patched means SEQ_LOG_VALS was set to 1.
>
> But ... pgbench -N doesn't use sequences at all, does it?
>
> Probably inserts into a table with a serial column would constitute a
> plausible real-world case.
>
D'oh! For some reason I thought pgbench has a sequence on the history
table, but clearly I was mistaken. There's another thinko, because after
inspecting pg_waldump output I realized "SEQ_LOG_VALS 1" actually logs
only every 2nd increment. So it should be "SEQ_LOG_VALS 0".
So I repeated the test fixing SEQ_LOG_VALS, and doing the pgbench with a
table like this:
create table test (a serial, b int);
and a script doing
insert into test (b) values (1);
The results look like this:
1) select nextval('s');
clients 1 4
------------------------------
master 39533 124998
patched 3748 9114
------------------------------
diff -91% -93%
2) insert into test (b) values (1);
clients 1 4
------------------------------
master 3718 9188
patched 3698 9209
------------------------------
diff 0% 0%
So the nextval() results are a bit worse, due to not caching 1/2 the
nextval calls. The -90% is roughly expected, due to generating about 32x
more WAL (and having to wait for commit).
But results for the more realistic insert workload are about the same as
before (i.e. no measurable difference). Also kinda expected, because
those transactions have to wait for WAL anyway.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-22 01:57 Tomas Vondra <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-22 01:57 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/19/21 04:03, Amit Kapila wrote:
> On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra
> <[email protected]> wrote:
>>
>> while working on logical decoding of sequences, I ran into an issue with
>> nextval() in a transaction that rolls back, described in [1]. But after
>> thinking about it a bit more (and chatting with Petr Jelinek), I think
>> this issue affects physical sync replication too.
>>
>> Imagine you have a primary <-> sync_replica cluster, and you do this:
>>
>> CREATE SEQUENCE s;
>>
>> -- shutdown the sync replica
>>
>> BEGIN;
>> SELECT nextval('s') FROM generate_series(1,50);
>> ROLLBACK;
>>
>> BEGIN;
>> SELECT nextval('s');
>> COMMIT;
>>
>> The natural expectation would be the COMMIT gets stuck, waiting for the
>> sync replica (which is not running), right? But it does not.
>>
>
> How about if we always WAL log the first sequence change in a transaction?
>
I've been thinking about doing something like this, but I think it would
not have any significant advantages compared to using "SEQ_LOG_VALS 0".
It would still have the same performance hit for plain nextval() calls,
and there's no measurable impact on simple workloads that already write
WAL in transactions even with SEQ_LOG_VALS 0.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-22 04:56 Fujii Masao <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Fujii Masao @ 2021-12-22 04:56 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 2021/12/22 10:57, Tomas Vondra wrote:
>
>
> On 12/19/21 04:03, Amit Kapila wrote:
>> On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra
>> <[email protected]> wrote:
>>>
>>> while working on logical decoding of sequences, I ran into an issue with
>>> nextval() in a transaction that rolls back, described in [1]. But after
>>> thinking about it a bit more (and chatting with Petr Jelinek), I think
>>> this issue affects physical sync replication too.
>>>
>>> Imagine you have a primary <-> sync_replica cluster, and you do this:
>>>
>>> CREATE SEQUENCE s;
>>>
>>> -- shutdown the sync replica
>>>
>>> BEGIN;
>>> SELECT nextval('s') FROM generate_series(1,50);
>>> ROLLBACK;
>>>
>>> BEGIN;
>>> SELECT nextval('s');
>>> COMMIT;
>>>
>>> The natural expectation would be the COMMIT gets stuck, waiting for the
>>> sync replica (which is not running), right? But it does not.
>>>
>>
>> How about if we always WAL log the first sequence change in a transaction?
>>
>
> I've been thinking about doing something like this, but I think it would not have any significant advantages compared to using "SEQ_LOG_VALS 0". It would still have the same performance hit for plain nextval() calls, and there's no measurable impact on simple workloads that already write WAL in transactions even with SEQ_LOG_VALS 0.
Just idea; if wal_level > minimal, how about making nextval_internal() (1) check whether WAL is replicated to sync standbys, up to the page lsn of the sequence, and (2) forcibly emit a WAL record if not replicated yet? The similar check is performed at the beginning of SyncRepWaitForLSN(), so probably we can reuse that code.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-22 12:11 Tomas Vondra <[email protected]>
parent: Fujii Masao <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-22 12:11 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; Amit Kapila <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/22/21 05:56, Fujii Masao wrote:
>
>
> On 2021/12/22 10:57, Tomas Vondra wrote:
>>
>>
>> On 12/19/21 04:03, Amit Kapila wrote:
>>> On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra
>>> <[email protected]> wrote:
>>>>
>>>> while working on logical decoding of sequences, I ran into an issue
>>>> with
>>>> nextval() in a transaction that rolls back, described in [1]. But after
>>>> thinking about it a bit more (and chatting with Petr Jelinek), I think
>>>> this issue affects physical sync replication too.
>>>>
>>>> Imagine you have a primary <-> sync_replica cluster, and you do this:
>>>>
>>>> CREATE SEQUENCE s;
>>>>
>>>> -- shutdown the sync replica
>>>>
>>>> BEGIN;
>>>> SELECT nextval('s') FROM generate_series(1,50);
>>>> ROLLBACK;
>>>>
>>>> BEGIN;
>>>> SELECT nextval('s');
>>>> COMMIT;
>>>>
>>>> The natural expectation would be the COMMIT gets stuck, waiting for the
>>>> sync replica (which is not running), right? But it does not.
>>>>
>>>
>>> How about if we always WAL log the first sequence change in a
>>> transaction?
>>>
>>
>> I've been thinking about doing something like this, but I think it
>> would not have any significant advantages compared to using
>> "SEQ_LOG_VALS 0". It would still have the same performance hit for
>> plain nextval() calls, and there's no measurable impact on simple
>> workloads that already write WAL in transactions even with
>> SEQ_LOG_VALS 0.
>
> Just idea; if wal_level > minimal, how about making nextval_internal()
> (1) check whether WAL is replicated to sync standbys, up to the page lsn
> of the sequence, and (2) forcibly emit a WAL record if not replicated
> yet? The similar check is performed at the beginning of
> SyncRepWaitForLSN(), so probably we can reuse that code.
>
Interesting idea, but I think it has a couple of issues :-(
1) We'd need to know the LSN of the last WAL record for any given
sequence, and we'd need to communicate that between backends somehow.
Which seems rather tricky to do without affecting performance.
2) SyncRepWaitForLSN() is used only in commit-like situations, and it's
a simple wait, not a decision to write more WAL. Environments without
sync replicas are affected by this too - yes, the data loss issue is not
there, but the amount of WAL is still increased.
IIRC sync_standby_names can change while a transaction is running, even
just right before commit, at which point we can't just go back in time
and generate WAL for sequences accessed earlier. But we still need to
ensure the sequence is properly replicated.
3) I don't think it'd actually reduce the amount of WAL records in
environments with many sessions (incrementing the same sequence). In
those cases the WAL (generated by in-progress xact from another session)
is likely to not be flushed, so we'd generate the extra WAL record. (And
if the other backends would need flush LSN of this new WAL record, which
would make it more likely they have to generate WAL too.)
So I don't think this would actually help much.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-22 17:50 Fujii Masao <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Fujii Masao @ 2021-12-22 17:50 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 2021/12/22 21:11, Tomas Vondra wrote:
> Interesting idea, but I think it has a couple of issues :-(
Thanks for the review!
> 1) We'd need to know the LSN of the last WAL record for any given sequence, and we'd need to communicate that between backends somehow. Which seems rather tricky to do without affecting performance.
How about using the page lsn for the sequence? nextval_internal() already uses that to check whether it's less than or equal to checkpoint redo location.
> 2) SyncRepWaitForLSN() is used only in commit-like situations, and it's a simple wait, not a decision to write more WAL. Environments without sync replicas are affected by this too - yes, the data loss issue is not there, but the amount of WAL is still increased.
How about reusing only a part of code in SyncRepWaitForLSN()? Attached is the PoC patch that implemented what I'm thinking.
> IIRC sync_standby_names can change while a transaction is running, even just right before commit, at which point we can't just go back in time and generate WAL for sequences accessed earlier. But we still need to ensure the sequence is properly replicated.
Yes. In the PoC patch, SyncRepNeedsWait() still checks sync_standbys_defined and uses SyncRepWaitMode. But they should not be checked nor used because their values can be changed on the fly, as you pointed out. Probably SyncRepNeedsWait() will need to be changed so that it doesn't use them.
> 3) I don't think it'd actually reduce the amount of WAL records in environments with many sessions (incrementing the same sequence). In those cases the WAL (generated by in-progress xact from another session) is likely to not be flushed, so we'd generate the extra WAL record. (And if the other backends would need flush LSN of this new WAL record, which would make it more likely they have to generate WAL too.)
With the PoC patch, only when previous transaction that executed nextval() and caused WAL record is aborted, subsequent nextval() generates additional WAL record. So this approach can reduce WAL volume than other approach?
In the PoC patch, to reduce WAL volume more, it might be better to make nextval_internal() update XactLastRecEnd and assign XID rather than emitting new WAL record, when SyncRepNeedsWait() returns true.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index 72bfdc07a4..38cd55b81a 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -37,6 +37,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_type.h"
+#include "replication/syncrep.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/smgr.h"
@@ -676,8 +677,9 @@ nextval_internal(Oid relid, bool check_permissions)
else
{
XLogRecPtr redoptr = GetRedoRecPtr();
+ XLogRecPtr pagelsn = PageGetLSN(page);
- if (PageGetLSN(page) <= redoptr)
+ if (pagelsn <= redoptr || SyncRepNeedsWait(pagelsn))
{
/* last update of seq was before checkpoint */
fetch = log = fetch + SEQ_LOG_VALS;
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index bdbc9ef844..589364cb58 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -130,6 +130,34 @@ static bool SyncRepQueueIsOrderedByLSN(int mode);
* ===========================================================
*/
+bool
+SyncRepNeedsWait(XLogRecPtr lsn)
+{
+ int mode;
+
+ if (!SyncRepRequested() ||
+ !((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+ return false;
+
+ mode = Min(SyncRepWaitMode, SYNC_REP_WAIT_FLUSH);
+
+ Assert(SHMQueueIsDetached(&(MyProc->syncRepLinks)));
+ Assert(WalSndCtl != NULL);
+
+ LWLockAcquire(SyncRepLock, LW_SHARED);
+ Assert(MyProc->syncRepState == SYNC_REP_NOT_WAITING);
+
+ if (!WalSndCtl->sync_standbys_defined ||
+ lsn <= WalSndCtl->lsn[mode])
+ {
+ LWLockRelease(SyncRepLock);
+ return false;
+ }
+
+ LWLockRelease(SyncRepLock);
+ return true;
+}
+
/*
* Wait for synchronous replication, if requested by user.
*
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 4266afde8b..b08f3f32d1 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -82,6 +82,7 @@ extern char *syncrep_parse_error_msg;
extern char *SyncRepStandbyNames;
/* called by user backend */
+extern bool SyncRepNeedsWait(XLogRecPtr lsn);
extern void SyncRepWaitForLSN(XLogRecPtr lsn, bool commit);
/* called at backend exit */
Attachments:
[text/plain] sequence_log.patch (2.2K, ../../[email protected]/2-sequence_log.patch)
download | inline diff:
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index 72bfdc07a4..38cd55b81a 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -37,6 +37,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_type.h"
+#include "replication/syncrep.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/smgr.h"
@@ -676,8 +677,9 @@ nextval_internal(Oid relid, bool check_permissions)
else
{
XLogRecPtr redoptr = GetRedoRecPtr();
+ XLogRecPtr pagelsn = PageGetLSN(page);
- if (PageGetLSN(page) <= redoptr)
+ if (pagelsn <= redoptr || SyncRepNeedsWait(pagelsn))
{
/* last update of seq was before checkpoint */
fetch = log = fetch + SEQ_LOG_VALS;
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index bdbc9ef844..589364cb58 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -130,6 +130,34 @@ static bool SyncRepQueueIsOrderedByLSN(int mode);
* ===========================================================
*/
+bool
+SyncRepNeedsWait(XLogRecPtr lsn)
+{
+ int mode;
+
+ if (!SyncRepRequested() ||
+ !((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+ return false;
+
+ mode = Min(SyncRepWaitMode, SYNC_REP_WAIT_FLUSH);
+
+ Assert(SHMQueueIsDetached(&(MyProc->syncRepLinks)));
+ Assert(WalSndCtl != NULL);
+
+ LWLockAcquire(SyncRepLock, LW_SHARED);
+ Assert(MyProc->syncRepState == SYNC_REP_NOT_WAITING);
+
+ if (!WalSndCtl->sync_standbys_defined ||
+ lsn <= WalSndCtl->lsn[mode])
+ {
+ LWLockRelease(SyncRepLock);
+ return false;
+ }
+
+ LWLockRelease(SyncRepLock);
+ return true;
+}
+
/*
* Wait for synchronous replication, if requested by user.
*
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 4266afde8b..b08f3f32d1 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -82,6 +82,7 @@ extern char *syncrep_parse_error_msg;
extern char *SyncRepStandbyNames;
/* called by user backend */
+extern bool SyncRepNeedsWait(XLogRecPtr lsn);
extern void SyncRepWaitForLSN(XLogRecPtr lsn, bool commit);
/* called at backend exit */
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-22 18:49 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-22 18:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 12/21/21 03:49, Tomas Vondra wrote:
> On 12/21/21 02:01, Tom Lane wrote:
>> Tomas Vondra <[email protected]> writes:
>>> OK, I did a quick test with two very simple benchmarks - simple select
>>> from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current
>>> master, patched means SEQ_LOG_VALS was set to 1.
>>
>> But ... pgbench -N doesn't use sequences at all, does it?
>>
>> Probably inserts into a table with a serial column would constitute a
>> plausible real-world case.
>>
>
> D'oh! For some reason I thought pgbench has a sequence on the history
> table, but clearly I was mistaken. There's another thinko, because after
> inspecting pg_waldump output I realized "SEQ_LOG_VALS 1" actually logs
> only every 2nd increment. So it should be "SEQ_LOG_VALS 0".
>
> So I repeated the test fixing SEQ_LOG_VALS, and doing the pgbench with a
> table like this:
>
> create table test (a serial, b int);
>
> and a script doing
>
> insert into test (b) values (1);
>
> The results look like this:
>
> 1) select nextval('s');
>
> clients 1 4
> ------------------------------
> master 39533 124998
> patched 3748 9114
> ------------------------------
> diff -91% -93%
>
>
> 2) insert into test (b) values (1);
>
> clients 1 4
> ------------------------------
> master 3718 9188
> patched 3698 9209
> ------------------------------
> diff 0% 0%
>
> So the nextval() results are a bit worse, due to not caching 1/2 the
> nextval calls. The -90% is roughly expected, due to generating about 32x
> more WAL (and having to wait for commit).
>
> But results for the more realistic insert workload are about the same as
> before (i.e. no measurable difference). Also kinda expected, because
> those transactions have to wait for WAL anyway.
>
Attached is a patch tweaking WAL logging - in wal_level=minimal we do
the same thing as now, in higher levels we log every sequence fetch.
After thinking about this a bit more, I think even the nextval workload
is not such a big issue, because we can set cache for the sequences.
Until now this had fairly limited impact, but it can significantly
reduce the performance drop caused by WAL-logging every sequence fetch.
I've repeated the nextval test on a different machine (the one I used
before is busy with something else), and the results look like this:
1) 1 client
cache 1 32 128
--------------------------------------
master 13975 14425 19886
patched 886 7900 18397
--------------------------------------
diff -94% -45% -7%
4) 4 clients
cache 1 32 128
-----------------------------------------
master 8338 12849 18248
patched 331 8124 18983
-----------------------------------------
diff -96% -37% 4%
So I think this makes it acceptable / manageable. Of course, this means
the values are much less monotonous (across backends), but I don't think
we really promised that. And I doubt anyone is really using sequences
like this (just nextval) in performance critical use cases.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[text/x-patch] 0001-WAL-log-individual-sequence-fetches-20211222.patch (1.9K, ../../[email protected]/2-0001-WAL-log-individual-sequence-fetches-20211222.patch)
download | inline diff:
From eeaa7cb36c69af048f0321e4883864ebe2542429 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 22 Dec 2021 03:18:46 +0100
Subject: [PATCH 1/6] WAL-log individual sequence fetches
---
src/backend/commands/sequence.c | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index 72bfdc07a4..0f309d0a4e 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -52,6 +52,9 @@
* We don't want to log each fetching of a value from a sequence,
* so we pre-log a few fetches in advance. In the event of
* crash we can lose (skip over) as many values as we pre-logged.
+ *
+ * We only pre-log fetches in wal_level=minimal. For higher levels we
+ * WAL-log every individual sequence increment, as if this was 0.
*/
#define SEQ_LOG_VALS 32
@@ -666,11 +669,18 @@ nextval_internal(Oid relid, bool check_permissions)
* WAL record to be written anyway, else replay starting from the
* checkpoint would fail to advance the sequence past the logged values.
* In this case we may as well fetch extra values.
+ *
+ * We only pre-log fetches in wal_level=minimal. For higher levels we
+ * WAL-log every individual sequence increment.
*/
if (log < fetch || !seq->is_called)
{
/* forced log to satisfy local demand for values */
- fetch = log = fetch + SEQ_LOG_VALS;
+ if (XLogIsNeeded())
+ fetch = log = fetch;
+ else
+ fetch = log = fetch + SEQ_LOG_VALS;
+
logit = true;
}
else
@@ -680,7 +690,11 @@ nextval_internal(Oid relid, bool check_permissions)
if (PageGetLSN(page) <= redoptr)
{
/* last update of seq was before checkpoint */
- fetch = log = fetch + SEQ_LOG_VALS;
+ if (XLogIsNeeded())
+ fetch = log = fetch;
+ else
+ fetch = log = fetch + SEQ_LOG_VALS;
+
logit = true;
}
}
--
2.31.1
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-22 19:00 Tomas Vondra <[email protected]>
parent: Fujii Masao <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Tomas Vondra @ 2021-12-22 19:00 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; Amit Kapila <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 12/22/21 18:50, Fujii Masao wrote:
>
>
> On 2021/12/22 21:11, Tomas Vondra wrote:
>> Interesting idea, but I think it has a couple of issues :-(
>
> Thanks for the review!
>
>> 1) We'd need to know the LSN of the last WAL record for any given
>> sequence, and we'd need to communicate that between backends somehow.
>> Which seems rather tricky to do without affecting performance.
>
> How about using the page lsn for the sequence? nextval_internal()
> already uses that to check whether it's less than or equal to checkpoint
> redo location.
>
Hmm, maybe.
>
>> 2) SyncRepWaitForLSN() is used only in commit-like situations, and
>> it's a simple wait, not a decision to write more WAL. Environments
>> without sync replicas are affected by this too - yes, the data loss
>> issue is not there, but the amount of WAL is still increased.
>
> How about reusing only a part of code in SyncRepWaitForLSN()? Attached
> is the PoC patch that implemented what I'm thinking.
>
>
>> IIRC sync_standby_names can change while a transaction is running,
>> even just right before commit, at which point we can't just go back in
>> time and generate WAL for sequences accessed earlier. But we still
>> need to ensure the sequence is properly replicated.
>
> Yes. In the PoC patch, SyncRepNeedsWait() still checks
> sync_standbys_defined and uses SyncRepWaitMode. But they should not be
> checked nor used because their values can be changed on the fly, as you
> pointed out. Probably SyncRepNeedsWait() will need to be changed so that
> it doesn't use them.
>
Right. I think the data loss with sync standby is merely a symptom, not
the root cause. We'd need to deduce the LSN for which to wait at commit.
>
>> 3) I don't think it'd actually reduce the amount of WAL records in
>> environments with many sessions (incrementing the same sequence). In
>> those cases the WAL (generated by in-progress xact from another
>> session) is likely to not be flushed, so we'd generate the extra WAL
>> record. (And if the other backends would need flush LSN of this new
>> WAL record, which would make it more likely they have to generate WAL
>> too.)
>
> With the PoC patch, only when previous transaction that executed
> nextval() and caused WAL record is aborted, subsequent nextval()
> generates additional WAL record. So this approach can reduce WAL volume
> than other approach?
> > In the PoC patch, to reduce WAL volume more, it might be better to make
> nextval_internal() update XactLastRecEnd and assign XID rather than
> emitting new WAL record, when SyncRepNeedsWait() returns true.
>
Yes, but I think there are other cases. For example the WAL might have
been generated by another backend, in a transaction that might be still
running. In which case I don't see how updating XactLastRecEnd in
nextval_internal would fix this, right?
I did some experiments with increasing CACHE for the sequence, and that
mostly eliminates the overhead. See the message I sent a couple minutes
ago. IMHO that's a reasonable solution for the tiny number of people
using nextval() in a way that'd be affected by this (i.e. without
writing anything else in the xact).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-23 14:42 Fujii Masao <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Fujii Masao @ 2021-12-23 14:42 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2021/12/23 3:49, Tomas Vondra wrote:
> Attached is a patch tweaking WAL logging - in wal_level=minimal we do the same thing as now, in higher levels we log every sequence fetch.
Thanks for the patch!
With the patch, I found that the regression test for sequences failed.
+ fetch = log = fetch;
This should be "log = fetch"?
On second thought, originally a sequence doesn't guarantee that the value already returned by nextval() will never be returned by subsequent nextval() after the server crash recovery. That is, nextval() may return the same value across crash recovery. Is this understanding right? For example, this case can happen if the server crashes after nextval() returned the value but before WAL for the sequence was flushed to the permanent storage. So it's not a bug that sync standby may return the same value as already returned in the primary because the corresponding WAL has not been replicated yet, isn't it?
BTW, if the returned value is stored in database, the same value is guaranteed not to be returned again after the server crash or by sync standby. Because in that case the WAL of the transaction storing that value is flushed and replicated.
> So I think this makes it acceptable / manageable. Of course, this means the values are much less monotonous (across backends), but I don't think we really promised that. And I doubt anyone is really using sequences like this (just nextval) in performance critical use cases.
I think that this approach is not acceptable to some users. So, if we actually adopt WAL-logging every sequence fetch, also how about exposing SEQ_LOG_VALS as reloption for a sequence? If so, those who want to log every sequence fetch can set this SEQ_LOG_VALS reloption to 0. OTOH, those who prefer the current behavior in spite of the risk we're discussing at this thread can set the reloption to 32 like it is for now, for example.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-23 18:50 Tomas Vondra <[email protected]>
parent: Fujii Masao <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-23 18:50 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 12/23/21 15:42, Fujii Masao wrote:
>
>
> On 2021/12/23 3:49, Tomas Vondra wrote:
>> Attached is a patch tweaking WAL logging - in wal_level=minimal we do
>> the same thing as now, in higher levels we log every sequence fetch.
>
> Thanks for the patch!
>
> With the patch, I found that the regression test for sequences failed.
>
> + fetch = log = fetch;
>
> This should be "log = fetch"?
>
> On second thought, originally a sequence doesn't guarantee that the
> value already returned by nextval() will never be returned by subsequent
> nextval() after the server crash recovery. That is, nextval() may return
> the same value across crash recovery. Is this understanding right? For
> example, this case can happen if the server crashes after nextval()
> returned the value but before WAL for the sequence was flushed to the
> permanent storage.
I think the important step is commit. We don't guarantee anything for
changes in uncommitted transactions. If you do nextval in a transaction
and the server crashes before the WAL gets flushed before COMMIT, then
yes, nextval may generate the same nextval again. But after commit that
is not OK - it must not happen.
> So it's not a bug that sync standby may return the same value as
> already returned in the primary because the corresponding WAL has not
> been replicated yet, isn't it?
>
No, I don't think so. Once the COMMIT happens (and gets confirmed by the
sync standby), it should be possible to failover to the sync replica
without losing any data in committed transaction. Generating duplicate
values is a clear violation of that.
IMHO the fact that we allow a transaction to commit (even just locally)
without flushing all the WAL it depends on is clearly a data loss bug.
> BTW, if the returned value is stored in database, the same value is
> guaranteed not to be returned again after the server crash or by sync
> standby. Because in that case the WAL of the transaction storing that
> value is flushed and replicated.
>
True, assuming the table is WAL-logged etc. I agree the issue may be
affecting a fairly small fraction of workloads, because most people use
sequences to generate data for inserts etc.
>> So I think this makes it acceptable / manageable. Of course, this
>> means the values are much less monotonous (across backends), but I
>> don't think we really promised that. And I doubt anyone is really
>> using sequences like this (just nextval) in performance critical use
>> cases.
>
> I think that this approach is not acceptable to some users. So, if we
> actually adopt WAL-logging every sequence fetch, also how about exposing
> SEQ_LOG_VALS as reloption for a sequence? If so, those who want to log
> every sequence fetch can set this SEQ_LOG_VALS reloption to 0. OTOH,
> those who prefer the current behavior in spite of the risk we're
> discussing at this thread can set the reloption to 32 like it is for
> now, for example.
>
I think it'd be worth explaining why you think it's not acceptable?
I've demonstrated the impact on regular workloads (with other changes
that write stuff to WAL) is not measurable, and enabling sequence
caching eliminates most of the overhead for the rare corner case
workloads if needed. It does generate a bit more WAL, but the sequence
WAL records are pretty tiny.
I'm opposed to adding relooptions that affect correctness - it just
seems like a bad idea to me. Moreover setting the CACHE for a sequence
does almost the same thing - if you set CACHE 32, we only generate WAL
once every 32 increments. The only difference is that this cache is not
shared between backends, so one backend will generate 1,2,3,... and
another backend will generate 33,34,35,... etc. I don't think that's a
problem, because if you want strictly monotonous / gap-less sequences
you can't use our sequences anyway. Yes, with short-lived backends this
may consume the sequences faster, but well - short-lived backends are
expensive anyway and overflowing bigserial is still unlikely.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-24 05:37 Kyotaro Horiguchi <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Kyotaro Horiguchi @ 2021-12-24 05:37 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]
At Thu, 23 Dec 2021 19:50:22 +0100, Tomas Vondra <[email protected]> wrote in
> On 12/23/21 15:42, Fujii Masao wrote:
> > On 2021/12/23 3:49, Tomas Vondra wrote:
> >> Attached is a patch tweaking WAL logging - in wal_level=minimal we do
> >> the same thing as now, in higher levels we log every sequence fetch.
> > Thanks for the patch!
> > With the patch, I found that the regression test for sequences failed.
> > + fetch = log = fetch;
> > This should be "log = fetch"?
> > On second thought, originally a sequence doesn't guarantee that the
> > value already returned by nextval() will never be returned by
> > subsequent nextval() after the server crash recovery. That is,
> > nextval() may return the same value across crash recovery. Is this
> > understanding right? For example, this case can happen if the server
> > crashes after nextval() returned the value but before WAL for the
> > sequence was flushed to the permanent storage.
>
> I think the important step is commit. We don't guarantee anything for
> changes in uncommitted transactions. If you do nextval in a
> transaction and the server crashes before the WAL gets flushed before
> COMMIT, then yes, nextval may generate the same nextval again. But
> after commit that is not OK - it must not happen.
I don't mean to stand on Fujii-san's side particularly, but it seems
to me sequences of RDBSs are not rolled back generally. Some googling
told me that at least Oracle (documented), MySQL, DB2 and MS-SQL
server doesn't rewind sequences at rollback, that is, sequences are
incremented independtly from transaction control. It seems common to
think that two nextval() calls for the same sequence must not return
the same value in any context.
> > So it's not a bug that sync standby may return the same value as
> > already returned in the primary because the corresponding WAL has not
> > been replicated yet, isn't it?
> >
>
> No, I don't think so. Once the COMMIT happens (and gets confirmed by
> the sync standby), it should be possible to failover to the sync
> replica without losing any data in committed transaction. Generating
> duplicate values is a clear violation of that.
So, strictly speaking, that is a violation of the constraint I
mentioned regardless whether the transaction is committed or
not. However we have technical limitations as below.
> IMHO the fact that we allow a transaction to commit (even just
> locally) without flushing all the WAL it depends on is clearly a data
> loss bug.
>
> > BTW, if the returned value is stored in database, the same value is
> > guaranteed not to be returned again after the server crash or by sync
> > standby. Because in that case the WAL of the transaction storing that
> > value is flushed and replicated.
> >
>
> True, assuming the table is WAL-logged etc. I agree the issue may be
> affecting a fairly small fraction of workloads, because most people
> use sequences to generate data for inserts etc.
It seems to me, from the fact that sequences are designed explicitly
untransactional and that behavior is widely adopted, the discussion
might be missing some significant use-cases. But there's a
possibility that the spec of sequence came from some technical
limitation in the past, but I'm not sure..
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-24 07:23 Tomas Vondra <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-24 07:23 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]
On 12/24/21 06:37, Kyotaro Horiguchi wrote:
> At Thu, 23 Dec 2021 19:50:22 +0100, Tomas Vondra <[email protected]> wrote in
>> On 12/23/21 15:42, Fujii Masao wrote:
>>> On 2021/12/23 3:49, Tomas Vondra wrote:
>>>> Attached is a patch tweaking WAL logging - in wal_level=minimal we do
>>>> the same thing as now, in higher levels we log every sequence fetch.
>>> Thanks for the patch!
>>> With the patch, I found that the regression test for sequences failed.
>>> + fetch = log = fetch;
>>> This should be "log = fetch"?
>>> On second thought, originally a sequence doesn't guarantee that the
>>> value already returned by nextval() will never be returned by
>>> subsequent nextval() after the server crash recovery. That is,
>>> nextval() may return the same value across crash recovery. Is this
>>> understanding right? For example, this case can happen if the server
>>> crashes after nextval() returned the value but before WAL for the
>>> sequence was flushed to the permanent storage.
>>
>> I think the important step is commit. We don't guarantee anything for
>> changes in uncommitted transactions. If you do nextval in a
>> transaction and the server crashes before the WAL gets flushed before
>> COMMIT, then yes, nextval may generate the same nextval again. But
>> after commit that is not OK - it must not happen.
>
> I don't mean to stand on Fujii-san's side particularly, but it seems
> to me sequences of RDBSs are not rolled back generally. Some googling
> told me that at least Oracle (documented), MySQL, DB2 and MS-SQL
> server doesn't rewind sequences at rollback, that is, sequences are
> incremented independtly from transaction control. It seems common to
> think that two nextval() calls for the same sequence must not return
> the same value in any context.
>
Yes, sequences are not rolled back on abort generally. That would
require much stricter locking, and that'd go against using sequences in
concurrent sessions.
But we're not talking about sequence rollback - we're talking about data
loss, caused by failure to flush WAL for a sequence. But that affects
the *current* code too, and to much greater extent.
Consider this:
BEGIN;
SELECT nextval('s') FROM generate_series(1,1000) s(i);
ROLLBACK; -- or crash of a different backend
BEGIN;
SELECT nextval('s');
COMMIT;
With the current code, this may easily lose the WAL, and we'll generate
duplicate values from the sequence. We pretty much ignore the COMMIT.
With the proposed change to WAL logging, that is not possible. The
COMMIT flushes enough WAL to prevent this issue.
So this actually makes this issue less severe.
Maybe I'm missing some important detail, though. Can you show an example
where the proposed changes make the issue worse?
>>> So it's not a bug that sync standby may return the same value as
>>> already returned in the primary because the corresponding WAL has not
>>> been replicated yet, isn't it?
>>>
>>
>> No, I don't think so. Once the COMMIT happens (and gets confirmed by
>> the sync standby), it should be possible to failover to the sync
>> replica without losing any data in committed transaction. Generating
>> duplicate values is a clear violation of that.
>
> So, strictly speaking, that is a violation of the constraint I
> mentioned regardless whether the transaction is committed or
> not. However we have technical limitations as below.
>
I don't follow. What violates what?
If the transaction commits (and gets a confirmation from sync replica),
the modified WAL logging prevents duplicate values. It does nothing for
uncommitted transactions. Seems like an improvement to me.
>> IMHO the fact that we allow a transaction to commit (even just
>> locally) without flushing all the WAL it depends on is clearly a data
>> loss bug.
>>
>>> BTW, if the returned value is stored in database, the same value is
>>> guaranteed not to be returned again after the server crash or by sync
>>> standby. Because in that case the WAL of the transaction storing that
>>> value is flushed and replicated.
>>>
>>
>> True, assuming the table is WAL-logged etc. I agree the issue may be
>> affecting a fairly small fraction of workloads, because most people
>> use sequences to generate data for inserts etc.
>
> It seems to me, from the fact that sequences are designed explicitly
> untransactional and that behavior is widely adopted, the discussion
> might be missing some significant use-cases. But there's a
> possibility that the spec of sequence came from some technical
> limitation in the past, but I'm not sure..
>
No idea. IMHO from the correctness / behavior point of view, the
modified logging is an improvement. The only issue is the additional
overhead, and I think the cache addresses that quite well.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-24 08:04 Kyotaro Horiguchi <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 2 replies; 84+ messages in thread
From: Kyotaro Horiguchi @ 2021-12-24 08:04 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]
At Fri, 24 Dec 2021 08:23:13 +0100, Tomas Vondra <[email protected]> wrote in
>
>
> On 12/24/21 06:37, Kyotaro Horiguchi wrote:
> > At Thu, 23 Dec 2021 19:50:22 +0100, Tomas Vondra
> > <[email protected]> wrote in
> >> On 12/23/21 15:42, Fujii Masao wrote:
> >>> On 2021/12/23 3:49, Tomas Vondra wrote:
> >>>> Attached is a patch tweaking WAL logging - in wal_level=minimal we do
> >>>> the same thing as now, in higher levels we log every sequence fetch.
> >>> Thanks for the patch!
> >>> With the patch, I found that the regression test for sequences failed.
> >>> + fetch = log = fetch;
> >>> This should be "log = fetch"?
> >>> On second thought, originally a sequence doesn't guarantee that the
> >>> value already returned by nextval() will never be returned by
> >>> subsequent nextval() after the server crash recovery. That is,
> >>> nextval() may return the same value across crash recovery. Is this
> >>> understanding right? For example, this case can happen if the server
> >>> crashes after nextval() returned the value but before WAL for the
> >>> sequence was flushed to the permanent storage.
> >>
> >> I think the important step is commit. We don't guarantee anything for
> >> changes in uncommitted transactions. If you do nextval in a
> >> transaction and the server crashes before the WAL gets flushed before
> >> COMMIT, then yes, nextval may generate the same nextval again. But
> >> after commit that is not OK - it must not happen.
> > I don't mean to stand on Fujii-san's side particularly, but it seems
> > to me sequences of RDBSs are not rolled back generally. Some googling
> > told me that at least Oracle (documented), MySQL, DB2 and MS-SQL
> > server doesn't rewind sequences at rollback, that is, sequences are
> > incremented independtly from transaction control. It seems common to
> > think that two nextval() calls for the same sequence must not return
> > the same value in any context.
> >
>
> Yes, sequences are not rolled back on abort generally. That would
> require much stricter locking, and that'd go against using sequences
> in concurrent sessions.
I thinks so.
> But we're not talking about sequence rollback - we're talking about
> data loss, caused by failure to flush WAL for a sequence. But that
> affects the *current* code too, and to much greater extent.
Ah, yes, I don't object to that aspect.
> Consider this:
>
> BEGIN;
> SELECT nextval('s') FROM generate_series(1,1000) s(i);
> ROLLBACK; -- or crash of a different backend
>
> BEGIN;
> SELECT nextval('s');
> COMMIT;
>
> With the current code, this may easily lose the WAL, and we'll
> generate duplicate values from the sequence. We pretty much ignore the
> COMMIT.
>
> With the proposed change to WAL logging, that is not possible. The
> COMMIT flushes enough WAL to prevent this issue.
>
> So this actually makes this issue less severe.
>
> Maybe I'm missing some important detail, though. Can you show an
> example where the proposed changes make the issue worse?
No. It seems to me improvoment at least from the current state, for
the reason you mentioned.
> >>> So it's not a bug that sync standby may return the same value as
> >>> already returned in the primary because the corresponding WAL has not
> >>> been replicated yet, isn't it?
> >>>
> >>
> >> No, I don't think so. Once the COMMIT happens (and gets confirmed by
> >> the sync standby), it should be possible to failover to the sync
> >> replica without losing any data in committed transaction. Generating
> >> duplicate values is a clear violation of that.
> > So, strictly speaking, that is a violation of the constraint I
> > mentioned regardless whether the transaction is committed or
> > not. However we have technical limitations as below.
> >
>
> I don't follow. What violates what?
>
> If the transaction commits (and gets a confirmation from sync
> replica), the modified WAL logging prevents duplicate values. It does
> nothing for uncommitted transactions. Seems like an improvement to me.
Sorry for the noise. I misunderstand that ROLLBACK is being changed to
rollback sequences.
> No idea. IMHO from the correctness / behavior point of view, the
> modified logging is an improvement. The only issue is the additional
> overhead, and I think the cache addresses that quite well.
Now I understand the story here.
I agree that the patch is improvment from the current behavior.
I agree that the overhead is eventually-nothing for WAL-emitting workloads.
Still, as Fujii-san concerns, I'm afraid that some people may suffer
the degradation the patch causes. I wonder it is acceptable to get
back the previous behavior by exposing SEQ_LOG_VALS itself or a
boolean to do that, as a 'not-recommended-to-use' variable.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-24 10:40 Tomas Vondra <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
1 sibling, 1 reply; 84+ messages in thread
From: Tomas Vondra @ 2021-12-24 10:40 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]
On 12/24/21 09:04, Kyotaro Horiguchi wrote:
>>> ...
>>> So, strictly speaking, that is a violation of the constraint I
>>> mentioned regardless whether the transaction is committed or
>>> not. However we have technical limitations as below.
>>>
>>
>> I don't follow. What violates what?
>>
>> If the transaction commits (and gets a confirmation from sync
>> replica), the modified WAL logging prevents duplicate values. It does
>> nothing for uncommitted transactions. Seems like an improvement to me.
>
> Sorry for the noise. I misunderstand that ROLLBACK is being changed to
> rollback sequences.
>
No problem, this part of the code is certainly rather confusing due to
several layers of caching and these WAL-logging optimizations.
>> No idea. IMHO from the correctness / behavior point of view, the
>> modified logging is an improvement. The only issue is the additional
>> overhead, and I think the cache addresses that quite well.
>
> Now I understand the story here.
>
> I agree that the patch is improvment from the current behavior.
> I agree that the overhead is eventually-nothing for WAL-emitting workloads.
>
OK, thanks.
> Still, as Fujii-san concerns, I'm afraid that some people may suffer
> the degradation the patch causes. I wonder it is acceptable to get
> back the previous behavior by exposing SEQ_LOG_VALS itself or a
> boolean to do that, as a 'not-recommended-to-use' variable.
>
Maybe, but what would such workload look like? Based on the tests I did,
such workload probably can't generate any WAL. The amount of WAL added
by the change is tiny, the regression is caused by having to flush WAL.
The only plausible workload I can think of is just calling nextval, and
the cache pretty much fixes that.
FWIW I plan to explore the idea of looking at sequence page LSN, and
flushing up to that position.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-27 20:24 Peter Eisentraut <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
1 sibling, 1 reply; 84+ messages in thread
From: Peter Eisentraut @ 2021-12-27 20:24 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; [email protected]; +Cc: [email protected]; [email protected]; [email protected]
On 24.12.21 09:04, Kyotaro Horiguchi wrote:
> Still, as Fujii-san concerns, I'm afraid that some people may suffer
> the degradation the patch causes. I wonder it is acceptable to get
> back the previous behavior by exposing SEQ_LOG_VALS itself or a
> boolean to do that, as a 'not-recommended-to-use' variable.
There is also the possibility of unlogged sequences if you want to avoid
the WAL logging and get higher performance.
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-28 01:39 Tomas Vondra <[email protected]>
parent: Peter Eisentraut <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Tomas Vondra @ 2021-12-28 01:39 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]
On 12/27/21 21:24, Peter Eisentraut wrote:
> On 24.12.21 09:04, Kyotaro Horiguchi wrote:
>> Still, as Fujii-san concerns, I'm afraid that some people may suffer
>> the degradation the patch causes. I wonder it is acceptable to get
>> back the previous behavior by exposing SEQ_LOG_VALS itself or a
>> boolean to do that, as a 'not-recommended-to-use' variable.
>
> There is also the possibility of unlogged sequences if you want to avoid
> the WAL logging and get higher performance.
But unlogged sequences are not supported:
test=# create unlogged sequence s;
ERROR: unlogged sequences are not supported
And even if we did, what would be the behavior after crash? For tables
we discard the contents, so for sequences we'd probably discard it too
and start from scratch? That doesn't seem particularly useful.
We could also write / fsync the sequence buffer, but that has other
downsides. But that's not implemented either, and it's certainly out of
scope for this patch.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-28 06:56 Fujii Masao <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Fujii Masao @ 2021-12-28 06:56 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]
On 2021/12/24 19:40, Tomas Vondra wrote:
> Maybe, but what would such workload look like? Based on the tests I did, such workload probably can't generate any WAL. The amount of WAL added by the change is tiny, the regression is caused by having to flush WAL.
>
> The only plausible workload I can think of is just calling nextval, and the cache pretty much fixes that.
Some users don't want to increase cache setting, do they? Because
- They may expect that setval() affects all subsequent nextval(). But if cache is set to greater than one, the value set by setval() doesn't affect other backends until they consumed all the cached sequence values.
- They may expect that the value returned from nextval() is basically increased monotonically. If cache is set to greater than one, subsequent nextval() can easily return smaller value than one returned by previous nextval().
- They may want to avoid "hole" of a sequence as much as possible, e.g., as far as the server is running normally. If cache is set to greater than one, such "hole" can happen even thought the server doesn't crash yet.
> FWIW I plan to explore the idea of looking at sequence page LSN, and flushing up to that position.
Sounds great, thanks!
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-28 08:28 Sascha Kuhl <[email protected]>
parent: Fujii Masao <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Sascha Kuhl @ 2021-12-28 08:28 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Kyotaro Horiguchi <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Sequence validation by step, in total is great. If the sequence is Familie
or professional, does it make sense to a have a total validation by an
expert. I can only say true by chi square Networks, but would a medical
opinion be an improvement?
Fujii Masao <[email protected]> schrieb am Di., 28. Dez. 2021,
07:56:
>
>
> On 2021/12/24 19:40, Tomas Vondra wrote:
> > Maybe, but what would such workload look like? Based on the tests I did,
> such workload probably can't generate any WAL. The amount of WAL added by
> the change is tiny, the regression is caused by having to flush WAL.
> >
> > The only plausible workload I can think of is just calling nextval, and
> the cache pretty much fixes that.
>
> Some users don't want to increase cache setting, do they? Because
>
> - They may expect that setval() affects all subsequent nextval(). But if
> cache is set to greater than one, the value set by setval() doesn't affect
> other backends until they consumed all the cached sequence values.
> - They may expect that the value returned from nextval() is basically
> increased monotonically. If cache is set to greater than one, subsequent
> nextval() can easily return smaller value than one returned by previous
> nextval().
> - They may want to avoid "hole" of a sequence as much as possible, e.g.,
> as far as the server is running normally. If cache is set to greater than
> one, such "hole" can happen even thought the server doesn't crash yet.
>
>
> > FWIW I plan to explore the idea of looking at sequence page LSN, and
> flushing up to that position.
>
> Sounds great, thanks!
>
> Regards,
>
> --
> Fujii Masao
> Advanced Computing Technology Center
> Research and Development Headquarters
> NTT DATA CORPORATION
>
>
>
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-28 08:50 Pavel Stehule <[email protected]>
parent: Sascha Kuhl <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Pavel Stehule @ 2021-12-28 08:50 UTC (permalink / raw)
To: Sascha Kuhl <[email protected]>; +Cc: Fujii Masao <[email protected]>; Tomas Vondra <[email protected]>; Kyotaro Horiguchi <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi
út 28. 12. 2021 v 9:28 odesílatel Sascha Kuhl <[email protected]>
napsal:
> Sequence validation by step, in total is great. If the sequence is Familie
> or professional, does it make sense to a have a total validation by an
> expert. I can only say true by chi square Networks, but would a medical
> opinion be an improvement?
>
Is it generated by boot or by a human?
> Fujii Masao <[email protected]> schrieb am Di., 28. Dez. 2021,
> 07:56:
>
>>
>>
>> On 2021/12/24 19:40, Tomas Vondra wrote:
>> > Maybe, but what would such workload look like? Based on the tests I
>> did, such workload probably can't generate any WAL. The amount of WAL added
>> by the change is tiny, the regression is caused by having to flush WAL.
>> >
>> > The only plausible workload I can think of is just calling nextval, and
>> the cache pretty much fixes that.
>>
>> Some users don't want to increase cache setting, do they? Because
>>
>> - They may expect that setval() affects all subsequent nextval(). But if
>> cache is set to greater than one, the value set by setval() doesn't affect
>> other backends until they consumed all the cached sequence values.
>> - They may expect that the value returned from nextval() is basically
>> increased monotonically. If cache is set to greater than one, subsequent
>> nextval() can easily return smaller value than one returned by previous
>> nextval().
>> - They may want to avoid "hole" of a sequence as much as possible, e.g.,
>> as far as the server is running normally. If cache is set to greater than
>> one, such "hole" can happen even thought the server doesn't crash yet.
>>
>>
>> > FWIW I plan to explore the idea of looking at sequence page LSN, and
>> flushing up to that position.
>>
>> Sounds great, thanks!
>>
>> Regards,
>>
>> --
>> Fujii Masao
>> Advanced Computing Technology Center
>> Research and Development Headquarters
>> NTT DATA CORPORATION
>>
>>
>>
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-28 08:53 Sascha Kuhl <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Sascha Kuhl @ 2021-12-28 08:53 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Fujii Masao <[email protected]>; Tomas Vondra <[email protected]>; Kyotaro Horiguchi <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Pavel Stehule <[email protected]> schrieb am Di., 28. Dez. 2021,
09:51:
> Hi
>
> út 28. 12. 2021 v 9:28 odesílatel Sascha Kuhl <[email protected]>
> napsal:
>
>> Sequence validation by step, in total is great. If the sequence is
>> Familie or professional, does it make sense to a have a total validation by
>> an expert. I can only say true by chi square Networks, but would a medical
>> opinion be an improvement?
>>
>
> Is it generated by boot or by a human?
>
I validation my family and Société, only when them Show me not their
Sekret, part of their truth. Works fine by a Boot level, as far as I can
detektei, without the Boot showing up 😉
>
>
>
>> Fujii Masao <[email protected]> schrieb am Di., 28. Dez. 2021,
>> 07:56:
>>
>>>
>>>
>>> On 2021/12/24 19:40, Tomas Vondra wrote:
>>> > Maybe, but what would such workload look like? Based on the tests I
>>> did, such workload probably can't generate any WAL. The amount of WAL added
>>> by the change is tiny, the regression is caused by having to flush WAL.
>>> >
>>> > The only plausible workload I can think of is just calling nextval,
>>> and the cache pretty much fixes that.
>>>
>>> Some users don't want to increase cache setting, do they? Because
>>>
>>> - They may expect that setval() affects all subsequent nextval(). But if
>>> cache is set to greater than one, the value set by setval() doesn't affect
>>> other backends until they consumed all the cached sequence values.
>>> - They may expect that the value returned from nextval() is basically
>>> increased monotonically. If cache is set to greater than one, subsequent
>>> nextval() can easily return smaller value than one returned by previous
>>> nextval().
>>> - They may want to avoid "hole" of a sequence as much as possible, e.g.,
>>> as far as the server is running normally. If cache is set to greater than
>>> one, such "hole" can happen even thought the server doesn't crash yet.
>>>
>>>
>>> > FWIW I plan to explore the idea of looking at sequence page LSN, and
>>> flushing up to that position.
>>>
>>> Sounds great, thanks!
>>>
>>> Regards,
>>>
>>> --
>>> Fujii Masao
>>> Advanced Computing Technology Center
>>> Research and Development Headquarters
>>> NTT DATA CORPORATION
>>>
>>>
>>>
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: sequences vs. synchronous replication
@ 2021-12-28 08:57 Pavel Stehule <[email protected]>
parent: Sascha Kuhl <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Stehule @ 2021-12-28 08:57 UTC (permalink / raw)
To: Sascha Kuhl <[email protected]>; +Cc: Fujii Masao <[email protected]>; Tomas Vondra <[email protected]>; Kyotaro Horiguchi <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
út 28. 12. 2021 v 9:53 odesílatel Sascha Kuhl <[email protected]>
napsal:
>
>
> Pavel Stehule <[email protected]> schrieb am Di., 28. Dez. 2021,
> 09:51:
>
>> Hi
>>
>> út 28. 12. 2021 v 9:28 odesílatel Sascha Kuhl <[email protected]>
>> napsal:
>>
>>> Sequence validation by step, in total is great. If the sequence is
>>> Familie or professional, does it make sense to a have a total validation by
>>> an expert. I can only say true by chi square Networks, but would a medical
>>> opinion be an improvement?
>>>
>>
>> Is it generated by boot or by a human?
>>
>
> I validation my family and Société, only when them Show me not their
> Sekret, part of their truth. Works fine by a Boot level, as far as I can
> detektei, without the Boot showing up 😉
>
don't spam this mailing list, please
Thank you
Pavel
>>
>>
>>> Fujii Masao <[email protected]> schrieb am Di., 28. Dez.
>>> 2021, 07:56:
>>>
>>>>
>>>>
>>>> On 2021/12/24 19:40, Tomas Vondra wrote:
>>>> > Maybe, but what would such workload look like? Based on the tests I
>>>> did, such workload probably can't generate any WAL. The amount of WAL added
>>>> by the change is tiny, the regression is caused by having to flush WAL.
>>>> >
>>>> > The only plausible workload I can think of is just calling nextval,
>>>> and the cache pretty much fixes that.
>>>>
>>>> Some users don't want to increase cache setting, do they? Because
>>>>
>>>> - They may expect that setval() affects all subsequent nextval(). But
>>>> if cache is set to greater than one, the value set by setval() doesn't
>>>> affect other backends until they consumed all the cached sequence values.
>>>> - They may expect that the value returned from nextval() is basically
>>>> increased monotonically. If cache is set to greater than one, subsequent
>>>> nextval() can easily return smaller value than one returned by previous
>>>> nextval().
>>>> - They may want to avoid "hole" of a sequence as much as possible,
>>>> e.g., as far as the server is running normally. If cache is set to greater
>>>> than one, such "hole" can happen even thought the server doesn't crash yet.
>>>>
>>>>
>>>> > FWIW I plan to explore the idea of looking at sequence page LSN, and
>>>> flushing up to that position.
>>>>
>>>> Sounds great, thanks!
>>>>
>>>> Regards,
>>>>
>>>> --
>>>> Fujii Masao
>>>> Advanced Computing Technology Center
>>>> Research and Development Headquarters
>>>> NTT DATA CORPORATION
>>>>
>>>>
>>>>
^ permalink raw reply [nested|flat] 84+ messages in thread
* Self contradictory examining on rel's baserestrictinfo
@ 2024-11-25 08:58 ro b <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: ro b @ 2024-11-25 08:58 UTC (permalink / raw)
To: pgsql-hackers <[email protected]>
Hi,
I committed the major features of self-contradictory examining patches.
I'd like to explain it in the following aspects.
1. Background
A few months ago, when i read source codes of B-tree in routine
_bt_preprocess_keys, i found that there are more contradictory
checking case we can add. I sent email to pgsql-hackers and
then community contributor replied me and told me someone had
already proposed this question. Thanks for taking the time
to address my question. After serveral conversations, i found
that we can do something more. We can place these jobs at planning time.
2. Under the frame
In practice, B-tree index is used widely; Binary operation expression
that Var op Const is often written; Also without data type coercion
and collation. So features above are supported.
There are several considerations 1) self-contradictory situation
is seldom appeared. 2) It's much expensive if spend too much effort
on it and find that rel is not self-contradictory. 3) The expression
evaluation mechanism will do the right things finally. So this is
a compromise.
3. Detailes
We will get the base restrictinfo list after deconstructed jointree.
1) The baserestrictinfo is a AND-implict list, that mean if one element
of the list is contradictory to others or self-contradictory the whole
rel is self-contradictory, then we can mark the rel is dummy.
2) The stragegy operators(>,>=,=,<,<=) of B-tree, some of them may be
contradictory to others. Another operator need to be considered is <>
the negator of =.
2.1) = against >, >=, =, <, <=, <>
2.2) >, >= against <, <=
3) We can eliminate the weaker restrictive operator if there is a more
restrictive operator (eg. x > 4 and x >=5, the sub-expression x > 4 will
be abandoned). This will simplify the subsequent examining.
A special situation we need to care about is if we encounter operator '='
and there is not contradictory, the operator '=' will dominate the others
operators.
4) We need to record down the details when we encounter the operator '<>'
and then check them oen by one if then operator '=' appear.
I use linear searching when the number of elements is less than or equal 4
or we can't find a support function to sort these elements.
If we find a support function and the number of elements is greater than 4,
sort these elements and use binary searching.
I choose number 4 here for two considerations. 1) the type size of storing
the const value multiply 4 match a cpu cache line usually. 2) In practice
we will not write down much not equal expressions generally.
5) NULL test
When IS NULL is set, it's contradictory to IS NOT NULL obviously.
And it's contradictory to operator that imply IS NOT NULL implictly.
If we find one of these operators ( >,>=,=,<,<=,<> ) and the expression with
the operator make sense (eg. NULL is not in expression) then we can recognize
this situation is contradictory to IS NULL test.
6) Boolean type
Boolean type is a little tricky, because it can evaluate with both of
test and opreator.
When we encounter the operator one of =, <> or expression is a single
Var or NOT Boolexpr with a single Var. We need to extract details to
make a test format (eg. x = true to x is true; x <> not false to x is false.)
When Var is set with IS_TRUE, this is contradictory to IS_NOT_TRUE,
IS_FALSE and IS_UNKNOWN.
When Var is set with IS_FALSE, this is contradictory to IS_TRUE,
IS_NOT_FALSE and IS_UNKNOWN.
When Var is set with IS_UNKNOWN, this is contradictory to IS_NOT_UNKNOWN,
IS_TRUE and IS_FALSE.
When the NULL test and self test of of boolean type are both appeared.
If IS NULL is set it's contradictory to IS TRUE, IS FALSE, IS NOT
UNKNOWN; If IS NOT NULL is set it's contradictory to IS UNKNOWN.
Finally we need check them carefully when strategy operator appear.
If we found boolean expression is written like x > true or x < false,
it's beyond the scope of the defined values of boolean type, the expression
does not make sense obviously.
If IS_UNKNOWN is set and it's contradictory to stragegy operator.
If IS_NOT_FALSE is set, that means the equivalent expression is (eg. x IS
NULL OR x IS TRUE). The NULL test will always fail with the strategy operator,
so we only need to test whether the sub-expression (x IS TRUE) is
contradictory with the strategy operator. If IS_NOT_TRUE is set, it's
similar to IS_NOT_FALSE.
7) Scalar array comparison expression
First we need to deconstruct the const array, figure out the null and non-null
elements.
If ALL flag is set and the Const contain NULL. we will get nothing (eg. x <=
ALL(array[56, null])), it's contradictory.
Then sort the non-null elements and eliminate any duplicates. We will give up
if we can't get the sorting support function.
1. Expression with operator <>
We can examine them one by one if ALL flag is set, it's similar to the common
expression (eg. x <> 2 and x <> 3);
If ANY flag is set and the number of sorted elements is only one, we can convert
the scalar array expression to a binary operator expression, but we can't do
anything else if there are more than one elements.
2. Expression with one of these operators >,>=,<,<=
We just need check the boundary element (eg. x > ALL(array[1,2,3]) it's equal
to x > 3) and convert expression to the equivalent binary operator expression.
3. Expression with operator =
We treat it as a binary operator expression if the number of sorted elements
is only one;
When the number of sorted elements is more than one.
It is contradictory if ALL flag is set.
If ANY flag is set, we need to postpone them before the collection of same type
expression is completed (eg. x in(1,3) and x in(1,5) we can extract the same
value 1 when we collected all of these expression over). Then we check
the intersection elements.
8) Row comparison expression
We need convert row comparison to nonconstructor.
The operator <> and = have already done in analysis phase. So we just need
to take care of one of operators <, <=, > and >=.
The basic rules for evaluating a row comparison expression is the row elements
are compared left-to-right, stopping as soon as an unequal or null pair
of elements is found.
Let us ignore the NULL that may appear in the row comparison expression
temporarily, the logic of deconstructing row comparison expression is like:
ROW(a1, a2, ..., an) >= ROW(b1, b2, ..., bn)
Normal format:
(a1 > b1)
OR (a1 = b1 AND a2 > b2)
OR (a1 = b1 AND a2 = b2 AND a3 > b3)
...
OR (a1 = b1 AND a2 = b2 AND ... AND an >= bn)
The difference between operators in row comparison expression is the more
restrictive operator is used in last sub-condition of every sub-OR expression
(eg. a3 > b3 the more restrictive > is used), but the operator of last
sub-condition (an >= bn) of the last sub-OR expression is according to the
operator of row comparison expression.
We get two regulations:
1. operator '=' is used in every sub-condition except the last one in
every sub-OR expression or the sub-OR expression had no equal sub-condition.
So we can record down the equal sub-condition with a list(i called it
sub-condition-list) and then we will iterate the list to generating
sub-conditions for subsequent sub-OR expression when we meet a new pair
of elements.
2. The more restrictive operator is used in last sub-condition of
every sub-OR expression except the last sub-OR expression. row expression's
operator weill be used int he last sub-condition of last sub-OR expression.
It's time to face the special cases now.
1. NULL value.
If either of this pair of elements is null, we can skip the evaluation of
subsequent elements, but if the NULL come out at first position then the
whole expression will supply nothing.
For example ROW(a,NULL,c) > (1,2,3)
normal format:
a > 1
or a = 1 and NULL > 2
or a = 1 and NULL = 2 and c > 3
The sub-OR expression doesn't make sense if NULL in sub-condition. So the
final converted expression is a > 1.
2. Pairs of elements are equal.
When they contain volatile functions, we will give up and just return back.
If they are not Const that mean element IS NOT NULL. We need add the NULL
test to the sub-condition-list for generating subsequent sub-OR expression.
It's need to ignore them if they are Const, so we don't add it to the
sub-condition-list.
We skip the current sub-OR expression generating, but if the pair of elements
come out at the last position we need be carefull.
If the operator of row comparison expression is a more restrictive operator,
we skip the last sub-OR expression generating, but if the operator is weaker
we need to generate the last sub-OR expression.
For examples:
ROW(a,b,c) > (1,b,3) ROW(a,2,c) > (1,2,3)
normal format: normal format:
a > 1 a > 1
or a = 1 and b > b or a = 1 and 2 > 2
or a = 1 and b = b and c > 3 or a = 1 and 2 = 2 and c > 3
The final expression is The final expression is
a > 1 a > 1
or a = 1 and b IS NOT NULL and c > 3 or a = 1 and c > 3
ROW(a,b,c) > (1,2,c) ROW(a,b,c) >= (1,2,c)
normal format: normal format:
a > 1 a > 1
or a = 1 and b > 2 or a = 1 and b > 2
or a = 1 and b = 2 and c > c or a = 1 and b = 2 and c >= c
The final expression is The final expression is
a > 1 a > 1
or a = 1 and b > 2 or a = 1 and b > 2
or a = 1 and b = 2 and c is not null
If case 1 and 2 are both appeared in row expression.
We will skip the subsequent pairs of elements when we meet the NULL.
So it just needs to be considered case 2 is in front of 1.
A special situation is all of pairs of elements are equal before we
meet NULL, according to the rule of case 2 we don't generate current
sub-OR expression if meet a pairs of elements these are equal.
So the nonconstructor is empty and we need a flag to know the result of
evaluation of the last pair of elements, then the result of row expression
is according to the flag.
3. Pairs of elements are Const but not equal.
Seeing the above normal format, there is no need to generate sub-OR
expression for the subsequent pairs of elements if we meet the pair of
elements they are Const but not equal.
But whether it's need to generate the current sub-OR expression or not is
according to the reulst of evaluation of current pair of elements.
If the reulst is positive we need to generate the sub-OR expression or
do nothing.
If the pair of elements come out at the first position, then the result
of row comparison expression is according to result of the pair of elements.
If the result is positive, the reuslt of row comparison expression is
positive.
Another situation is all of pairs of elements are equal before we meet
the not equal pair of elements, it's similar to what we discussed in case 2.
We will get a OR-expression finally. Then we can do the examining jobs and
prune the OR-expression, we only keep the output expression that is a simple
expression. The OR-format is not better than row expression for subsequent
processing. If we need watch the output OR-expression we should add macro
DEBUG_ROW_DECODE when configure our environment.
9) OR-expression
We need postpone OR-expression if we meet it. Because we can push other
expressions these have the same semantic level with OR-expression down into
the OR-expression. Then we can check every sub-expression with the pushed
down details.
we will abandon the sub-expression if we find the sub-expression is
contradictory.
when all of the sub-expressions are contradictory that mean the OR-expression
is contradictory.
A special situation is once we find a sub-expression is const true and then
the OR-expression is const true.
For example: x > 10 and (x < 9 or y > 10) in the expression we push x > 10
down into the OR-expression, then we will find the sub-expression x < 9 is
contradictory to x > 10 and x < 9 will be abandoned, so the final expression
is x > 10 AND y > 10.
10) EC situation
We may get a clause match the EC mechanism after we finished the all jobs.
For example: x > 10 and (x < 9 or y = 10) we will get the expression y = 10
match the EC, so we need add it into the EC.
11) The phase 2 examining
As we know in the case 10, if we add a ec and we may get a contradictory case
after the generate_base_implied_equalities invoked. So we need to recheck the
rel when rel's baserestrictinfo added clauses.
For example: select * from self_a x join self_a y on x.a= y.a where x.a in
(2,3,45) and x.a in (45, 78) and y.a <> 45
we will get the intersection element 45 and convert
(x.a in (2,3,45) and x.a in (45, 78)) to x.a = 45, then add it into EC.
We will get the generated clause y.a = 45 after
generate_base_implied_equalities invoked, it's contradictory to y.a <>45,
so alias y is self-contradictory finally.
Best regards
Attachments:
[application/octet-stream] patch.diff (78.6K, ../../TYCPR01MB6093E2A06F3F8CF8DF17B5CE852E2@TYCPR01MB6093.jpnprd01.prod.outlook.com/3-patch.diff)
download | inline diff:
From e94d26d35ea8e6ff9c69576bb31e10248eb29805 Mon Sep 17 00:00:00 2001
From: wq <[email protected]>
Date: Mon, 25 Nov 2024 16:27:34 +0800
Subject: [PATCH] Self contradictory examining on rel's baserestrictinfo
---
src/backend/optimizer/plan/Makefile | 3 +-
src/backend/optimizer/plan/contradictory.c | 2640 ++++++++++++++++++++
src/backend/optimizer/plan/initsplan.c | 16 +-
src/backend/optimizer/plan/meson.build | 1 +
src/backend/optimizer/plan/planmain.c | 4 +
src/backend/utils/cache/lsyscache.c | 39 +
src/include/nodes/pathnodes.h | 7 +
src/include/optimizer/planmain.h | 2 +
src/include/utils/lsyscache.h | 3 +
9 files changed, 2711 insertions(+), 4 deletions(-)
create mode 100644 src/backend/optimizer/plan/contradictory.c
diff --git a/src/backend/optimizer/plan/Makefile b/src/backend/optimizer/plan/Makefile
index 80ef162e48..36ba23664a 100644
--- a/src/backend/optimizer/plan/Makefile
+++ b/src/backend/optimizer/plan/Makefile
@@ -20,6 +20,7 @@ OBJS = \
planmain.o \
planner.o \
setrefs.o \
- subselect.o
+ subselect.o \
+ contradictory.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/optimizer/plan/contradictory.c b/src/backend/optimizer/plan/contradictory.c
new file mode 100644
index 0000000000..a58a825b32
--- /dev/null
+++ b/src/backend/optimizer/plan/contradictory.c
@@ -0,0 +1,2640 @@
+/*-------------------------------------------------------------------------
+ *
+ * initsplan.c
+ * Target list, qualification, joininfo initialization routines
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/optimizer/plan/contradictory.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "nodes/nodeFuncs.h"
+#include "nodes/makefuncs.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/typcache.h"
+#include "utils/array.h"
+#include "access/nbtree.h"
+#include "lib/qunique.h"
+#include "common/int.h"
+#include "optimizer/restrictinfo.h"
+#include "optimizer/paths.h"
+
+/*
+ * Self-contradictory definations
+ *
+ */
+#define DEFAULT_ELEMS_NUM 4
+#define STRATEGY_NOT_EQUAL (BTMaxStrategyNumber + 1) /* operator '<>' */
+
+typedef enum ClauseCxtType
+{
+ CCT_DEL,
+ CCT_CONSTT, /* Evaluated expr is const and true. */
+ CCT_CONSTF, /* Evaluated expr is const and false. */
+ CCT_REPLACE
+}ClauseCxtType;
+
+#define NT_MARK (1<<7)
+#define NT_IS_NULL (1<<IS_NULL)
+#define NT_IS_NOT_NULL (1<<IS_NOT_NULL)
+
+/* Boolean test definations. */
+#define BT_MARK (1<<7)
+#define BT_IS_TRUE (1<<IS_TRUE)
+#define BT_IS_NOT_TRUE (1<<IS_NOT_TRUE)
+#define BT_IS_FALSE (1<<IS_FALSE)
+#define BT_IS_NOT_FALSE (1<<IS_NOT_FALSE)
+#define BT_IS_UNKNOWN (1<<IS_UNKNOWN)
+#define BT_IS_NOT_UNKNOWN (1<<IS_NOT_UNKNOWN)
+
+typedef struct ConstVal
+{
+ bool push_down;
+ uint16 strategy;
+ uint32 idx;
+ Expr *expr;
+ Datum *value;
+ FmgrInfo op_func;
+}ConstVal;
+
+typedef struct SAEqualDatElem
+{
+ uint32 idx;
+ int elem_nums;
+ Expr *expr;
+ Datum *elem_values;
+}SAEqualDatElem;
+
+typedef struct SortArrayContext
+{
+ FmgrInfo sortproc;
+ Oid collation;
+ Oid elmtype;
+ int16 elmlen;
+ bool elmbyval;
+ char elmalign;
+ bool reverse;
+} SortArrayContext;
+
+typedef struct SortNeConstCxt
+{
+ FmgrInfo cmp_proc;
+ Oid collation;
+ bool reverse;
+} SortNeConstCxt;
+
+typedef struct SAEqualDat
+{
+ RegProcedure opfuncid;
+ SortArrayContext sortcxt;
+ List *upper_elems;
+ List *elemnents; /* Type of SAEqualDatElem. */
+}SAEqualDat;
+
+typedef struct NeConstDat
+{
+ uint32 idx;
+ Datum *val;
+}NeConstDat;
+
+typedef struct NeConstVal
+{
+ SortNeConstCxt sncc;
+ bool can_sort;
+ bool sorted;
+ FmgrInfo op_func;
+ int size;
+ int capacity;
+ NeConstDat *ncds;
+ struct NeConstVal *upper_ncv;
+}NeConstVal;
+
+typedef struct VarattInfo
+{
+ uint8 stratcount;
+ uint8 null_test;
+ uint8 bool_test;
+ uint16 atidx;
+ Oid collid;
+ Oid type;
+ NeConstVal ne_const;
+ SAEqualDat saop;
+ ConstVal st_table[BTMaxStrategyNumber];
+ struct VarattInfo *next;
+}VarattInfo;
+
+typedef struct RowCompElem
+{
+ Oid opid;
+ Oid stricter_opid;
+ Expr *contact;
+}RowCompElem;
+
+typedef struct ExtractedAttris
+{
+ bool addOp;
+ AttrNumber min_attr;
+ VarattInfo **array_vi;
+ VarattInfo *att_lst;
+ VarattInfo *last;
+ struct ExtractedAttris *upper_ea;
+}ExtractedAttris;
+
+typedef struct ClauseContext
+{
+ ClauseCxtType cct;
+ uint32 idx;
+ Expr *origin;
+ void *cxt;
+}ClauseContext;
+
+typedef struct ClauseCxtWapper
+{
+ List *clausecxt; /* Type of ClauseContext. */
+}ClauseCxtWapper;
+
+typedef struct AttrOpFamily
+{
+ Oid negator;
+ Oid opfamily;
+}AttrOpFamily;
+
+typedef struct AttrOpFamilySet
+{
+ AttrOpFamily elems[DEFAULT_ELEMS_NUM];
+ struct AttrOpFamilySet *next;
+}AttrOpFamilySet;
+
+
+typedef struct AttrsDetailsCache
+{
+ AttrNumber min_attr;
+ AttrNumber max_attr;
+ int len_set; /* The number of current AttrOpFamilySet elements used. */
+ AttrOpFamily **atop_array;
+ AttrOpFamilySet *atop_set;
+}AttrsDetailsCache;
+
+typedef struct OpcInfoElement
+{
+ Oid opcid;
+ Oid opcfamily;
+ Oid opcintype;
+}OpcInfoElement;
+
+typedef struct SortedOpclassInfo
+{
+ int n_members;
+ OpcInfoElement **opcies;
+}SortedOpclassInfo;
+
+typedef struct SelfExamContext
+{
+ SortedOpclassInfo sopcinfo;
+ AttrsDetailsCache attrsdc;
+ ClauseContext *ccfree;
+ VarattInfo *vifree;
+}SelfExamContext;
+
+#define EXAMINE_ABORT(x) \
+do{ \
+ ret = x; \
+ goto finished; \
+}while(0)
+
+extern void check_mergejoinable(RestrictInfo *restrictinfo);
+extern bool contain_volatile_functions(Node *clause);
+void examine_self_contradictory_rels_phase1(PlannerInfo *root);
+void examine_self_contradictory_rels_phase2(PlannerInfo *root);
+
+static inline ClauseContext* get_clause_cxt_internal(SelfExamContext *sec)
+{
+ ClauseContext *ccxt;
+ if (sec->ccfree)
+ {
+ ccxt = sec->ccfree;
+ sec->ccfree = (ClauseContext*) ccxt->cxt;
+ ccxt->cxt = NULL;
+ ccxt->origin = NULL;
+ }
+ else
+ ccxt = palloc0(sizeof(ClauseContext));
+
+ return ccxt;
+}
+
+static inline void free_clause_cxt_internal(SelfExamContext *sec, ClauseContext *cc)
+{
+ if (sec->ccfree)
+ {
+ cc->cxt = sec->ccfree;
+ sec->ccfree = cc;
+ }
+ else
+ {
+ cc->cxt = NULL;
+ sec->ccfree = cc;
+ }
+}
+
+static inline VarattInfo* get_attinfo_cxt_internal(SelfExamContext *sec)
+{
+ VarattInfo *varatt;
+ if (sec->vifree)
+ {
+ varatt = sec->vifree;
+ sec->vifree = varatt->next;
+ varatt->next = NULL;
+ }
+ else
+ varatt = palloc0(sizeof(VarattInfo));
+
+ return varatt;
+}
+
+static inline void free_attinfo_cxt_internal(SelfExamContext *sec, VarattInfo *atinfo)
+{
+ if (sec->vifree)
+ {
+ atinfo->next = sec->vifree;
+ sec->vifree = atinfo;
+ }
+ else
+ {
+ atinfo->next = NULL;
+ sec->vifree = atinfo;
+ }
+}
+
+
+static inline AttrOpFamily* get_atop_element_internal(AttrsDetailsCache *attrsdc)
+{
+ AttrOpFamilySet *atops;
+ if (attrsdc->len_set == DEFAULT_ELEMS_NUM)
+ {
+ atops = palloc0(sizeof(AttrOpFamilySet));
+ atops->next = attrsdc->atop_set;
+ attrsdc->atop_set = atops;
+ attrsdc->len_set = 0;
+ }
+ else
+ attrsdc->atop_set->elems[attrsdc->len_set].negator = InvalidOid;
+ return attrsdc->atop_set->elems + attrsdc->len_set++;
+}
+
+static int compare_scalararray_elements(const void *a, const void *b, void *arg)
+{
+ Datum da = *((const Datum *) a);
+ Datum db = *((const Datum *) b);
+ SortArrayContext *cxt = (SortArrayContext *) arg;
+ int32 compare;
+
+ compare = DatumGetInt32(FunctionCall2Coll(&cxt->sortproc,
+ cxt->collation,
+ da, db));
+ if (cxt->reverse)
+ INVERT_COMPARE_RESULT(compare);
+ return compare;
+}
+
+static int sort_scalararray_elements(SortArrayContext *cxt,
+ Oid intype,
+ Oid opfamily,
+ Oid collid,
+ bool reverse,
+ Datum *elems,
+ int nelems)
+{
+ RegProcedure cmp_proc;
+
+ if (nelems <= 1)
+ return nelems; /* No work to do */
+ /*
+ * Look up the appropriate comparison function in the opfamily.
+ *
+ * Note: it's possible that this would fail, if the opfamily is
+ * incomplete, but it seems quite unlikely that an opfamily would omit
+ * non-cross-type support functions for any datatype that it supports at
+ * all.
+ */
+ cmp_proc = get_opfamily_proc(opfamily,
+ intype,
+ intype,
+ BTORDER_PROC);
+ if (!RegProcedureIsValid(cmp_proc))
+ return 0; /* No support function, give up, do nothing. */
+
+ /* Sort the array elements */
+ fmgr_info(cmp_proc, &cxt->sortproc);
+ cxt->collation = collid;
+ cxt->reverse = reverse;
+ qsort_arg((void *) elems, nelems, sizeof(Datum),
+ compare_scalararray_elements, (void *) cxt);
+
+ /* Now scan the sorted elements and remove duplicates */
+ return qunique_arg(elems, nelems, sizeof(Datum),
+ compare_scalararray_elements, cxt);
+}
+
+static void extract_opc_info(int nums, Oid opcid, Oid opcfamily, Oid opcintype, void *infocxt)
+{
+ int i;
+ OpcInfoElement *opcele;
+ SortedOpclassInfo *sopcinfo = (SortedOpclassInfo*) infocxt;
+ if (sopcinfo->n_members == 0)
+ {
+ sopcinfo->opcies = palloc0(nums * (sizeof(OpcInfoElement*) + sizeof(OpcInfoElement)));
+ opcele = (OpcInfoElement*) (sopcinfo->opcies + nums);
+ for (i = 0; i < nums; i++)
+ sopcinfo->opcies[i] = opcele + i;
+ }
+
+ sopcinfo->opcies[sopcinfo->n_members]->opcfamily = opcfamily;
+ sopcinfo->opcies[sopcinfo->n_members]->opcid = opcid;
+ sopcinfo->opcies[sopcinfo->n_members]->opcintype = opcintype;
+ sopcinfo->n_members++;
+ Assert(sopcinfo->n_members <= nums);
+}
+
+static int opc_element_cmp(const void *p1, const void *p2)
+{
+ Oid v1 = (*((const OpcInfoElement **) p1))->opcintype;
+ Oid v2 = (*((const OpcInfoElement **) p2))->opcintype;
+
+ return pg_cmp_u32(v1, v2);
+}
+
+static Oid binary_search_opc_info(SortedOpclassInfo *sopi, Oid inputtype)
+{
+ Assert(sopi != NULL);
+ if (sopi->n_members > 0)
+ {
+ int i, l, r, res;
+ l = 0;
+ r = sopi->n_members - 1;
+ while (l <= r)
+ {
+ i = (l + r) / 2;
+ res = pg_cmp_u32(sopi->opcies[i]->opcintype, inputtype);
+ if (res == 0)
+ {
+ return sopi->opcies[i]->opcfamily;
+ }
+ else if (res < 0)
+ l = i + 1;
+ else
+ r = i - 1;
+ }
+ }
+ return InvalidOid;
+}
+
+static int ne_const_arg_cmp(const void *a, const void *b, void *arg)
+{
+ Datum da = *(((const NeConstDat *) a)->val);
+ Datum db = *(((const NeConstDat *) b)->val);
+ SortNeConstCxt *cncc = (SortNeConstCxt *) arg;
+ int32 compare;
+
+ compare = DatumGetInt32(FunctionCall2Coll(&cncc->cmp_proc,
+ cncc->collation,
+ da, db));
+ return compare;
+}
+
+static inline void init_ne_const(NeConstVal *nv)
+{
+ if (nv->capacity == 0)
+ {
+ nv->ncds = palloc0(DEFAULT_ELEMS_NUM * sizeof(NeConstDat));
+ nv->size = 0;
+ nv->capacity = DEFAULT_ELEMS_NUM;
+ }
+}
+
+static inline void insert_ne_const(NeConstVal *nv, uint32 idx, Datum *dat)
+{
+ if (nv->size == nv->capacity) {
+ nv->capacity *= 2;
+ nv->ncds = repalloc(nv->ncds, nv->capacity * sizeof(NeConstDat));
+ }
+ nv->ncds[nv->size].idx = idx;
+ nv->ncds[nv->size].val = dat;
+ nv->size++;
+ nv->sorted = false;
+}
+
+/* binary_search_ne_const
+ * Note: caller need to check number of elements in NeConstVal.
+ *
+ */
+static bool binary_search_ne_const(NeConstVal *nv, Datum *dat)
+{
+ int i, l, r, res;
+ NeConstDat ncd;
+ ncd.val = dat;
+ l = 0;
+ r = nv->size - 1;
+ while (l <= r)
+ {
+ i = (l + r) / 2;
+ res = ne_const_arg_cmp(&nv->ncds[i], &ncd, (void *) &nv->sncc);
+ if (res == 0)
+ {
+ return true;
+ }
+ else if (res < 0)
+ l = i + 1;
+ else
+ r = i - 1;
+ }
+ return false;
+}
+
+
+static bool search_ne_const(NeConstVal *nv, Oid collid, Datum *val)
+{
+ int i;
+ bool result;
+
+ if (nv->upper_ncv)
+ {
+ if (search_ne_const(nv->upper_ncv, collid, val))
+ return true;
+ }
+
+ if (nv->size == 0)
+ return false;
+
+ /* If there is no support function or size is small less than 4, use
+ * linear searching.
+ */
+ if (!nv->can_sort || nv->size <= DEFAULT_ELEMS_NUM)
+ {
+ for (i = 0; i < nv->size;i++)
+ {
+ result = DatumGetBool(FunctionCall2Coll(&nv->op_func,
+ collid,
+ *val,
+ *nv->ncds[i].val));
+ if (!result)
+ return true;
+ }
+ return false;
+ }
+
+ if (nv->can_sort && !nv->sorted)
+ {
+ qsort_arg((void *) nv->ncds, nv->size, sizeof(NeConstDat),
+ ne_const_arg_cmp, (void *) &nv->sncc);
+ nv->sorted = true;
+ }
+
+ if (binary_search_ne_const(nv, val))
+ return true;
+ return false;
+}
+
+static VarattInfo* get_varatt_info_internal(SelfExamContext *sec, ExtractedAttris *ea, Var *var)
+{
+ VarattInfo *attinfo, *attupper;
+ ConstVal *constv;
+ ExtractedAttris *up_ea;
+ NeConstVal **up_ncv;
+ int idx, i;
+ idx = var->varattno - ea->min_attr;
+ if (ea->array_vi[idx] == NULL)
+ {
+ attinfo = get_attinfo_cxt_internal(sec);
+ if (ea->att_lst)
+ {
+ attinfo->next = ea->att_lst;
+ ea->att_lst = attinfo;
+ }
+ else
+ ea->att_lst = ea->last = attinfo;
+ ea->array_vi[idx] = attinfo;
+
+ attinfo->stratcount = 0;
+ attinfo->null_test = 0;
+ attinfo->bool_test = 0;
+ attinfo->atidx = idx;
+ attinfo->collid = var->varcollid;
+ attinfo->type = var->vartype;
+ attinfo->ne_const.size = 0;
+ attinfo->ne_const.upper_ncv = NULL;
+ attinfo->saop.elemnents = NIL;
+ attinfo->saop.upper_elems = NIL;
+ for(i = 0; i < BTMaxStrategyNumber; i++)
+ attinfo->st_table[i].strategy = InvalidStrategy;
+
+ /* Copy push-down attribute particulars. */
+ up_ea = ea->upper_ea;
+ up_ncv = &attinfo->ne_const.upper_ncv;
+ while(up_ea)
+ {
+ if (up_ea->array_vi[idx])
+ {
+ attupper = up_ea->array_vi[idx];
+
+ if (attinfo->null_test == 0 && attupper->null_test != 0)
+ attinfo->null_test = attupper->null_test;
+ if (attinfo->bool_test == 0 && attupper->bool_test != 0)
+ attinfo->bool_test = attupper->bool_test;
+ if (!*up_ncv && attupper->ne_const.size != 0)
+ {
+ *up_ncv = &attupper->ne_const;
+ up_ncv = &(*up_ncv)->upper_ncv;
+ }
+ if (attinfo->stratcount == 0 && attupper->stratcount != 0)
+ {
+ attinfo->stratcount = attupper->stratcount;
+ for(i = 0; i < BTMaxStrategyNumber; i++)
+ {
+ constv = &attupper->st_table[i];
+ if (constv->strategy != InvalidStrategy)
+ {
+ memcpy(&attinfo->st_table[i], constv, sizeof(ConstVal));
+ attinfo->st_table[i].push_down = true;
+ }
+ }
+ }
+ if (attinfo->saop.upper_elems == NIL && attupper->saop.elemnents != NIL)
+ attinfo->saop.upper_elems = attupper->saop.elemnents;
+ }
+ up_ea = up_ea->upper_ea;
+ }
+ }
+ else
+ attinfo = ea->array_vi[idx];
+ return attinfo;
+}
+
+static bool deconstruct_nulltest_attr(Node *ntnode,
+ NullTestType ntt,
+ SelfExamContext *sec,
+ ExtractedAttris *ea)
+{
+ if (IsA(ntnode, Var))
+ {
+ VarattInfo *att_elem;
+ att_elem = get_varatt_info_internal(sec, ea, (Var *) ntnode);
+
+ if (ntt == IS_NULL)
+ att_elem->null_test = att_elem->null_test | NT_MARK | NT_IS_NULL;
+ else
+ att_elem->null_test = att_elem->null_test | NT_MARK | NT_IS_NOT_NULL;
+ if (att_elem->null_test == (NT_MARK | NT_IS_NULL | NT_IS_NOT_NULL))
+ {
+ return false;
+ }
+ }
+ else if (IsA(ntnode, ArrayExpr))
+ {
+ ListCell *l;
+ foreach(l, ((ArrayExpr *) ntnode)->elements)
+ {
+ if(!deconstruct_nulltest_attr((Node *) lfirst(l), ntt, sec, ea))
+ return false;
+ }
+ }
+
+ return true;
+}
+
+static bool add_binary_op_clause_info(VarattInfo *att_elem,
+ Datum *vals,
+ int nums,
+ uint32 idx,
+ uint16 strat,
+ RegProcedure opfuncid,
+ Expr *expr,
+ SelfExamContext *sec,
+ ClauseCxtWapper *ccw)
+{
+ ConstVal *constval;
+ ClauseContext *ccxt;
+ AttrOpFamily *atop;
+ Oid cmp_proc;
+ bool result;
+
+ if (strat == STRATEGY_NOT_EQUAL)
+ {
+ if (att_elem->st_table[BTEqualStrategyNumber-1].strategy != InvalidStrategy)
+ {
+ constval = &(att_elem->st_table[BTEqualStrategyNumber-1]);
+ result = DatumGetBool(FunctionCall2Coll(&constval->op_func,
+ att_elem->collid,
+ *vals,
+ *(constval->value)));
+ if (result)
+ return false;
+ }
+
+ if (att_elem->ne_const.size == 0)
+ {
+ init_ne_const(&att_elem->ne_const);
+ att_elem->ne_const.can_sort = false;
+ att_elem->ne_const.sorted = false;
+ fmgr_info(opfuncid, &(att_elem->ne_const.op_func));
+ atop = sec->attrsdc.atop_array[att_elem->atidx];
+ cmp_proc = get_opfamily_proc(atop->opfamily,
+ att_elem->type,
+ att_elem->type,
+ BTORDER_PROC);
+ if (!RegProcedureIsValid(cmp_proc))
+ {
+ /* No support function, use linear searching. */
+ att_elem->ne_const.can_sort = false;
+ }
+ else
+ {
+ att_elem->ne_const.sncc.reverse = false;
+ fmgr_info(cmp_proc, &att_elem->ne_const.sncc.cmp_proc);
+ att_elem->ne_const.sncc.collation = att_elem->collid;
+ att_elem->ne_const.can_sort = true;
+ }
+ }
+ insert_ne_const(&att_elem->ne_const, idx, vals);
+ }
+ else if (strat <= BTMaxStrategyNumber)
+ {
+ Assert(strat >= BTLessStrategyNumber);
+ constval = &(att_elem->st_table[strat-1]);
+ if (constval->strategy == InvalidStrategy)
+ {
+ att_elem->stratcount++;
+ constval->strategy = strat;
+ fmgr_info(opfuncid, &(constval->op_func));
+ constval->value = vals;
+ constval->expr = expr;
+ constval->push_down = false;
+ constval->idx = idx;
+ }
+ else
+ {
+ result = DatumGetBool(FunctionCall2Coll(&constval->op_func,
+ att_elem->collid,
+ *vals,
+ *(constval->value)));
+ /* Repeat operator = but the consts ain't equal ( eg. x=2 and x=3 ). */
+ if (strat == BTEqualStrategyNumber)
+ {
+ if (!result)
+ return false;
+ if (constval->push_down)
+ {
+ constval->value = vals;
+ constval->expr = expr;
+ constval->push_down = false;
+ constval->idx = idx;
+ }
+ else if (IsA(expr, OpExpr))
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ else if (result)
+ {
+ /* Current Expr is more restrict, so keep it. */
+ if (!constval->push_down && IsA(constval->expr, OpExpr))
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = constval->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ constval->value = vals;
+ constval->expr = expr;
+ constval->push_down = false;
+ constval->idx = idx;
+ }
+ else
+ {
+ /* Current Expr is less restrict, so abandon it. */
+ if (constval->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = idx;
+ ccxt->cxt = copyObjectImpl(constval->expr);
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ constval->push_down = false;
+ constval->expr = ccxt->cxt;
+ constval->idx = idx;
+ }
+ else if (IsA(expr, OpExpr))
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ }
+ }
+ return true;
+}
+
+static bool self_contradictory_element(SelfExamContext *sec,
+ ClauseCxtWapper *ccw,
+ VarattInfo *att_elem)
+{
+ ConstVal *leftcv, *rightcv;
+ ClauseContext *ccxt;
+ int i;
+ uint8 btest;
+ bool result;
+ bool macthed;
+
+ if (att_elem->bool_test & BT_MARK)
+ {
+ btest = att_elem->bool_test & ~BT_MARK;
+ if (((btest & BT_IS_TRUE) && (btest & (BT_IS_NOT_TRUE | BT_IS_FALSE | BT_IS_UNKNOWN)))
+ || ((btest & BT_IS_FALSE) && (btest &(BT_IS_TRUE | BT_IS_NOT_FALSE | BT_IS_UNKNOWN)))
+ || ((btest & BT_IS_UNKNOWN) && (btest &(BT_IS_NOT_UNKNOWN | BT_IS_TRUE | BT_IS_FALSE))))
+ {
+ return true;
+ }
+ }
+
+ if (att_elem->null_test == (NT_MARK | NT_IS_NULL)
+ && (att_elem->stratcount > 0
+ || att_elem->ne_const.size != 0
+ || att_elem->ne_const.upper_ncv))
+ return true;
+
+ if ((att_elem->bool_test & BT_MARK) && (att_elem->null_test & NT_MARK))
+ {
+ if (att_elem->null_test == (NT_MARK | NT_IS_NULL)
+ && (btest & (BT_IS_TRUE | BT_IS_FALSE | BT_IS_NOT_UNKNOWN)))
+ return true;
+ else if (att_elem->null_test == (NT_MARK | NT_IS_NOT_NULL)
+ && (btest & BT_IS_UNKNOWN))
+ return true;
+ }
+
+ /* If operator '=' and '<>' are both appeared. */
+ if (att_elem->st_table[BTEqualStrategyNumber-1].strategy != 0
+ && (att_elem->ne_const.size != 0 || att_elem->ne_const.upper_ncv))
+ {
+ leftcv = &att_elem->st_table[BTEqualStrategyNumber-1];
+
+ if (search_ne_const(&att_elem->ne_const, att_elem->collid, leftcv->value))
+ return true;
+ /* If they are not contradictory, eliminate the not equalities. */
+ for (i = 0; i < att_elem->ne_const.size; i++)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = att_elem->ne_const.ncds[i].idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ att_elem->ne_const.size = 0;
+ }
+
+ /* Try to keep only one of <, <= */
+ if (att_elem->st_table[BTLessStrategyNumber - 1].strategy != 0
+ && att_elem->st_table[BTLessEqualStrategyNumber - 1].strategy != 0)
+ {
+ leftcv = &att_elem->st_table[BTLessStrategyNumber - 1];
+ rightcv = &att_elem->st_table[BTLessEqualStrategyNumber - 1];
+ result = DatumGetBool(FunctionCall2Coll(&(rightcv->op_func),
+ att_elem->collid,
+ *(leftcv->value),
+ *(rightcv->value)));
+ if (result)
+ {
+ rightcv->strategy = InvalidStrategy;
+ if (leftcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = rightcv->idx;
+ ccxt->cxt = copyObjectImpl(leftcv->expr);
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ leftcv->push_down = false;
+ leftcv->expr = ccxt->cxt;
+ leftcv->idx = rightcv->idx;
+ }
+ else if (!rightcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = rightcv->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ else
+ {
+ leftcv->strategy = InvalidStrategy;
+ if (rightcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = leftcv->idx;
+ ccxt->cxt = copyObjectImpl(rightcv->expr);
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ rightcv->push_down = false;
+ rightcv->expr = ccxt->cxt;
+ rightcv->idx = leftcv->idx;
+ }
+ else if (!leftcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = leftcv->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ att_elem->stratcount--;
+ }
+
+ /* try to keep only one of >, >= */
+ if (att_elem->st_table[BTGreaterStrategyNumber - 1].strategy != 0
+ && att_elem->st_table[BTGreaterEqualStrategyNumber - 1].strategy != 0)
+ {
+ leftcv = &att_elem->st_table[BTGreaterStrategyNumber - 1];
+ rightcv = &att_elem->st_table[BTGreaterEqualStrategyNumber - 1];
+ result = DatumGetBool(FunctionCall2Coll(&(rightcv->op_func),
+ att_elem->collid,
+ *(leftcv->value),
+ *(rightcv->value)));
+ if (result)
+ {
+ rightcv->strategy = InvalidStrategy;
+ if (leftcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = rightcv->idx;
+ ccxt->cxt = copyObjectImpl(leftcv->expr);
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ leftcv->push_down = false;
+ leftcv->expr = ccxt->cxt;
+ leftcv->idx = rightcv->idx;
+ }
+ else if (!rightcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = rightcv->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ else
+ {
+ leftcv->strategy = InvalidStrategy;
+ if (rightcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = leftcv->idx;
+ ccxt->cxt = copyObjectImpl(rightcv->expr);
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ rightcv->push_down = false;
+ rightcv->expr = ccxt->cxt;
+ rightcv->idx = leftcv->idx;
+ }
+ else if (!leftcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = leftcv->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ att_elem->stratcount--;
+ }
+
+ if (att_elem->type == BOOLOID && att_elem->stratcount > 0)
+ {
+ /*
+ * Beyond the scope of the defined values of boolean type,
+ * obviously it does not make sense.
+ */
+ if (att_elem->st_table[BTGreaterStrategyNumber - 1].strategy != 0)
+ {
+ leftcv = &att_elem->st_table[BTGreaterStrategyNumber - 1];
+ result = DatumGetBool(FunctionCall2Coll(&leftcv->op_func,
+ att_elem->collid,
+ BoolGetDatum(true),
+ *leftcv->value));
+ if (!result)
+ return true;
+ }
+ if (att_elem->st_table[BTLessStrategyNumber - 1].strategy != 0)
+ {
+ leftcv = &att_elem->st_table[BTLessStrategyNumber - 1];
+ result = DatumGetBool(FunctionCall2Coll(&leftcv->op_func,
+ att_elem->collid,
+ BoolGetDatum(false),
+ *leftcv->value));
+ if (!result)
+ return true;
+ }
+
+ if (att_elem->bool_test & BT_MARK)
+ {
+ btest = att_elem->bool_test & ~BT_MARK;
+ if (btest & BT_IS_UNKNOWN)
+ return true;
+ else
+ {
+ bool isvalue;
+ macthed = true;
+ /* If IS_NOT_FALSE is set, that mean the equivalent expression is
+ * (x IS NULL OR x IS TRUE). The NULL test will always fail with
+ * the strategy operator, so we only need to test whether the
+ * sub-expression (x IS TRUE) is contradictory to the strategy
+ * operator. If IS_NOT_TRUE is set, it's similar to IS_NOT_FALSE.
+ */
+ if (btest & (BT_IS_TRUE| BT_IS_NOT_FALSE))
+ isvalue = true;
+ else if (btest & (BT_IS_FALSE| BT_IS_NOT_TRUE))
+ isvalue = false;
+ else
+ macthed = false;
+ if (macthed)
+ {
+ for (i = BTMaxStrategyNumber; --i >= 0;)
+ {
+ leftcv = &att_elem->st_table[i];
+ if (leftcv->strategy == InvalidStrategy)
+ continue;
+ result = DatumGetBool(FunctionCall2Coll(&leftcv->op_func,
+ att_elem->collid,
+ BoolGetDatum(isvalue),
+ *leftcv->value));
+ if (!result)
+ return true;
+ }
+ }
+ }
+ }
+ }
+
+ if (att_elem->stratcount <=1)
+ return false;
+
+ if (att_elem->st_table[BTEqualStrategyNumber-1].strategy != 0)
+ {
+ leftcv = &att_elem->st_table[BTEqualStrategyNumber - 1];
+ for (i = BTMaxStrategyNumber; --i >= 0;)
+ {
+ rightcv = &att_elem->st_table[i];
+ if (rightcv->strategy == InvalidStrategy || i == (BTEqualStrategyNumber - 1))
+ continue;
+ result = DatumGetBool(FunctionCall2Coll(&(rightcv->op_func),
+ att_elem->collid,
+ *(leftcv->value),
+ *(rightcv->value)));
+ if (!result)
+ return true;
+ /* The element had be dominated by operator =, so abandon it. */
+ if (leftcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = rightcv->idx;
+ ccxt->cxt = copyObjectImpl(leftcv->expr);
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ leftcv->push_down = false;
+ leftcv->expr = ccxt->cxt;
+ leftcv->idx = rightcv->idx;
+ }
+ else if (!rightcv->push_down)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = rightcv->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ rightcv->strategy = InvalidStrategy;
+ }
+ /* The operator = has dominated others, so no need to check other case. */
+ return false;
+ }
+
+ /* Detect the contradictory case that like x <4 and x >10. */
+ macthed = true;
+ if (att_elem->st_table[BTGreaterStrategyNumber - 1].strategy != 0)
+ {
+ rightcv = &att_elem->st_table[BTGreaterStrategyNumber - 1];
+ if (att_elem->st_table[BTLessStrategyNumber - 1].strategy != 0)
+ leftcv = &att_elem->st_table[BTLessStrategyNumber - 1];
+ else if (att_elem->st_table[BTLessEqualStrategyNumber - 1].strategy != 0)
+ leftcv = &att_elem->st_table[BTLessEqualStrategyNumber - 1];
+ else
+ macthed = false;
+ }
+ else if (att_elem->st_table[BTLessStrategyNumber - 1].strategy != 0)
+ {
+ rightcv = &att_elem->st_table[BTLessStrategyNumber - 1];
+ if (att_elem->st_table[BTGreaterStrategyNumber - 1].strategy != 0)
+ leftcv = &att_elem->st_table[BTGreaterStrategyNumber - 1];
+ else if (att_elem->st_table[BTGreaterEqualStrategyNumber - 1].strategy != 0)
+ leftcv = &att_elem->st_table[BTGreaterEqualStrategyNumber - 1];
+ else
+ macthed = false;
+ }
+ else if (att_elem->st_table[BTLessEqualStrategyNumber - 1].strategy != 0
+ && att_elem->st_table[BTGreaterEqualStrategyNumber - 1].strategy != 0)
+ {
+ rightcv = &att_elem->st_table[BTGreaterEqualStrategyNumber - 1];
+ leftcv = &att_elem->st_table[BTLessEqualStrategyNumber - 1];
+ }
+ else
+ macthed = false;
+ if (macthed)
+ {
+ result = DatumGetBool(FunctionCall2Coll(&(rightcv->op_func),
+ att_elem->collid,
+ *(leftcv->value),
+ *(rightcv->value)));
+ if (!result)
+ return true;
+ }
+
+ return false;
+}
+
+static bool check_eq_sa_op_elemnet(Datum *value, VarattInfo *att_elem)
+{
+ ConstVal *constval;
+ int i;
+ bool result;
+ uint8 btest;
+ bool isvalue, macthed;
+
+ if (att_elem->null_test == (NT_MARK | NT_IS_NULL))
+ return true;
+
+ if (att_elem->bool_test & BT_MARK)
+ {
+ btest = att_elem->bool_test & ~BT_MARK;
+
+ if (btest & BT_IS_UNKNOWN)
+ return true;
+
+ macthed = true;
+ if (btest & (BT_IS_TRUE| BT_IS_NOT_FALSE))
+ isvalue = true;
+ else if (btest & (BT_IS_FALSE| BT_IS_NOT_TRUE))
+ isvalue = false;
+ else
+ macthed = false;
+
+ if (macthed && DatumGetBool(*value) != isvalue)
+ return true;
+ }
+
+ if (search_ne_const(&att_elem->ne_const, att_elem->collid, value))
+ return true;
+ if (att_elem->stratcount > 0)
+ {
+ for (i = BTMaxStrategyNumber; --i >= 0;)
+ {
+ constval = &att_elem->st_table[i];
+ if (constval->strategy == InvalidStrategy)
+ continue;
+ result = DatumGetBool(FunctionCall2Coll(&(constval->op_func),
+ att_elem->collid,
+ *(value),
+ *(constval->value)));
+ if (!result)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * self_contradictory_saop
+ * The Scalar array expr of any-equality expression are postponed here
+ * to examine them (eg. x in (1,3) and x in (3,5) and x = 1 ).
+ * Extract intersection elements and report to the caller if there is
+ * no intersection element or check intersection elements one by one
+ * and eliminate it if it's self-contradictory. All of them are
+ * self-contradictory report it.
+ * Rebuilt the saop finally.
+ */
+static bool self_contradictory_saop(SelfExamContext *sec,
+ ClauseCxtWapper *ccw,
+ VarattInfo *att_elem)
+{
+ /* Scalar array expr of single attribute. */
+ ScalarArrayOpExpr *saopexpr;
+ SAEqualDatElem *saout;
+ ClauseContext *ccxt;
+ ListCell *lc;
+ Expr *newclause;
+ Const *con;
+ int i, j, initial_nums;
+
+ if (att_elem->saop.elemnents == NIL)
+ return false;
+
+ if (att_elem->saop.upper_elems != NIL)
+ att_elem->saop.elemnents = list_concat(att_elem->saop.elemnents, att_elem->saop.upper_elems);
+
+ saout = linitial(att_elem->saop.elemnents);
+ initial_nums = saout->elem_nums;
+ if (list_length(att_elem->saop.elemnents) > 1)
+ {
+ /* Initialize saout with the first one, merge the rest of elemnents
+ * one by one and extract the intersection element at the same time.
+ * Update the saout with merged result.
+ */
+ for_each_from(lc, att_elem->saop.elemnents, 1)
+ {
+ SAEqualDatElem *sade;
+ int nelems_dups;
+
+ sade =lfirst(lc);
+ nelems_dups = 0;
+ for (i = 0, j = 0; i < saout->elem_nums && j < sade->elem_nums;)
+ {
+ Datum *oelem = saout->elem_values + i,
+ *nelem = sade->elem_values + j;
+ int res = compare_scalararray_elements(oelem, nelem, &att_elem->saop.sortcxt);
+ if (res == 0)
+ {
+ saout->elem_values[nelems_dups++] = *oelem;
+ i++;
+ j++;
+ }
+ else if (res < 0)
+ i++;
+ else /* res > 0 */
+ j++;
+ }
+ if (nelems_dups == 0)
+ {
+ att_elem->saop.elemnents = NIL;
+ return true; /* No intersection element, so report it. */
+ }
+ else
+ saout->elem_nums = nelems_dups;
+ }
+ }
+
+ /* Examine the rest of intersection elements one by one. */
+ for(i = 0; i < saout->elem_nums;)
+ {
+ if (check_eq_sa_op_elemnet(&saout->elem_values[i], att_elem))
+ {
+ if (i < saout->elem_nums - 1)
+ memmove(saout->elem_values + i, saout->elem_values + i + 1,
+ sizeof(Datum) * (saout->elem_nums - i - 1));
+ saout->elem_nums--;
+ }
+ else
+ i++;
+ }
+ if (0 == saout->elem_nums)
+ {
+ att_elem->saop.elemnents = NIL;
+ return true;
+ }
+
+ /*
+ * Convert the intersection elements to a new clause and delete orignal
+ * elements.
+ */
+ saopexpr = (ScalarArrayOpExpr*)saout->expr;
+ if (saout->elem_nums == 1)
+ {
+ con = makeConst(att_elem->saop.sortcxt.elmtype,
+ -1, /* typmod -1 is OK for all cases */
+ InvalidOid,
+ att_elem->saop.sortcxt.elmlen,
+ saout->elem_values[0],
+ false,
+ att_elem->saop.sortcxt.elmbyval);
+ con->location = -1;
+ newclause = make_opclause(saopexpr->opno,
+ BOOLOID,
+ false,
+ linitial(saopexpr->args),
+ (Expr*)con,
+ InvalidOid,
+ saopexpr->inputcollid);
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = saout->idx;
+ ccxt->cxt = newclause;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ add_binary_op_clause_info(att_elem,
+ &saout->elem_values[0],
+ 1,
+ saout->idx,
+ BTEqualStrategyNumber,
+ att_elem->saop.opfuncid,
+ newclause,
+ sec,
+ ccw);
+ }
+ else if (saout->elem_nums != initial_nums)
+ {
+ /* It's need to rebuilt the first saop if there are multi-saop,
+ * and delete others saop later.
+ */
+ ArrayType *arrval = construct_array(saout->elem_values,
+ saout->elem_nums,
+ att_elem->saop.sortcxt.elmtype,
+ att_elem->saop.sortcxt.elmlen,
+ att_elem->saop.sortcxt.elmbyval,
+ att_elem->saop.sortcxt.elmalign);
+ con = lsecond(saopexpr->args);
+ con->constvalue = PointerGetDatum(arrval);
+
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = saout->idx;
+ ccxt->cxt = saopexpr;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+
+ }
+
+ /* The push-down elements have finished their jobs, so truncate them. */
+ if (att_elem->saop.upper_elems != NIL)
+ { att_elem->saop.elemnents = list_truncate(att_elem->saop.elemnents, \
+ list_length(att_elem->saop.elemnents) - list_length(att_elem->saop.upper_elems));
+ }
+
+ /* Since the whole elements except first one have be merged into first one,
+ * it's time to delete them from the orignal clause list.
+ */
+ for_each_from(lc, att_elem->saop.elemnents, 1)
+ {
+ saout =lfirst(lc);
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_DEL;
+ ccxt->idx = saout->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ /* It's need to truncate the list, it maybe push down to
+ * compare with lower level ScalarArrayOpExpr, since the
+ * whole elements except first one have already merged.
+ */
+ att_elem->saop.elemnents = list_truncate(att_elem->saop.elemnents, 1);
+ return false;
+}
+
+static bool self_contradictory_check(SelfExamContext *sec, ExtractedAttris *ea, ClauseCxtWapper *ccw)
+{
+ VarattInfo *attinfo;
+ if (!ea->addOp)
+ return false;
+ attinfo = ea->att_lst;
+ while(attinfo)
+ {
+ if (self_contradictory_element(sec, ccw, attinfo))
+ return true;
+
+ if (self_contradictory_saop(sec, ccw, attinfo))
+ return true;
+ attinfo = attinfo->next;
+ }
+ return false;
+}
+
+/*
+ * extract_binary_op_clause_info
+ * Extract binary operation clause information
+ *
+ * In practice, B-tree index is used widely; Binary operation
+ * expr that Var op Const is often written; Also without
+ * data type coercion and collation. So this routine support
+ * above features.
+ * There are several consideration 1) self-contradictory
+ * rel seldom appeared. 2) Invoker will get upset if spend
+ * too much effort on it and find that rel is not
+ * self-contradictory (eg. strict function sqrt, expr like
+ * sqrt(x) > 2 and sqrt(x) < 4 ). 3) Finally the expression
+ * evaluation mechanism will do the right things. So this is
+ * a compromise.
+ */
+static bool extract_binary_op_clause_info(void *expr,
+ Node *lop,
+ Node *rop,
+ uint32 idx,
+ Oid opno,
+ Oid collid,
+ SelfExamContext *sec,
+ ExtractedAttris *ea,
+ ClauseCxtWapper *ccw)
+{
+ VarattInfo *attinfo;
+ Node *leftop, *rightop;
+ AttrOpFamily *atop;
+ Expr *newclause;
+ Oid left_type, right_type;
+ Oid varcollid;
+ Oid opid, opfamily;
+ RegProcedure opfuncid;
+ StrategyNumber strat;
+ uint16 attidx;
+
+ if (!(IsA(lop, Const) || IsA(rop, Const)))
+ return true;
+
+ if (IsA(lop, Const))
+ {
+ opid = get_commutator(opno);
+ if (!OidIsValid(opid))
+ return true;
+ leftop = rop;
+ rightop = lop;
+ }
+ else
+ {
+ opid = opno;
+ leftop = lop;
+ rightop = rop;
+ }
+ opfuncid = get_opcode(opid);
+ Assert(OidIsValid(opfuncid));
+
+ if (((Const *) rightop)->constisnull)
+ return true;
+
+ left_type = InvalidOid;
+ right_type = ((Const *) rightop)->consttype;
+
+ if (IsA(expr, ScalarArrayOpExpr))
+ {
+ Oid typInput, typelem;
+ getTypeInputInfo(right_type, &typInput, &typelem);
+ right_type = typelem;
+ }
+
+ if (leftop && IsA(leftop, RelabelType))
+ {
+ left_type = ((RelabelType *) leftop)->resulttype;
+ varcollid = ((RelabelType *) leftop)->resultcollid;
+ leftop = (Node*)((RelabelType *) leftop)->arg;
+ }
+
+ if (IsA(leftop, Var))
+ {
+ if (((Var *) leftop)->varattno > sec->attrsdc.max_attr)
+ elog(ERROR, "bogus relation attribute");
+ attidx = ((Var *) leftop)->varattno - sec->attrsdc.min_attr;
+ }
+ else /* Don't spend much effort for other types, so skip it */
+ return true;
+
+ if (!OidIsValid(left_type))
+ {
+ left_type = ((Var *) leftop)->vartype;
+ varcollid = ((Var *) leftop)->varcollid;
+ }
+
+ if (left_type != right_type)
+ return true;
+ if (collid != varcollid)
+ return true;
+
+ /* Search attribute opfamily cache, be happy to use it if find. */
+ if (sec->attrsdc.atop_array[attidx])
+ {
+ atop = sec->attrsdc.atop_array[attidx];
+ if (!OidIsValid(atop->opfamily))
+ return true;
+ opfamily = atop->opfamily;
+ }
+ else
+ {
+ atop = get_atop_element_internal(&sec->attrsdc);
+ sec->attrsdc.atop_array[attidx] = atop;
+
+ opfamily = binary_search_opc_info(&sec->sopcinfo, left_type);
+ if (!OidIsValid(opfamily))
+ {
+ atop->opfamily = InvalidOid;
+ return true;
+ }
+ atop->opfamily = opfamily;
+ }
+
+ if (!OidIsValid(atop->negator) && atop->negator == opid)
+ strat = STRATEGY_NOT_EQUAL;
+ else
+ {
+ strat = get_op_opfamily_strategy(opid, opfamily);
+ if (strat == 0)
+ {
+ /* Perhaps it is a <> operator. See if it has a negator that is in an opfamily. */
+ Oid op_negator = get_negator(opid);
+ if (!OidIsValid(op_negator))
+ return true;
+ if (BTEqualStrategyNumber == get_op_opfamily_strategy(op_negator, opfamily))
+ {
+ atop->negator = opid;
+ strat = STRATEGY_NOT_EQUAL;
+ }
+ else
+ return true;
+ }
+ }
+
+ /* If it's scalar array expr, deconstruct the const values. */
+ if (unlikely(IsA(expr, ScalarArrayOpExpr)))
+ {
+ ArrayType *arrayval;
+ int num_elems;
+ Datum *elem_values, *one_elem;
+ bool *elem_nulls;
+ int num_nonnulls;
+ int j;
+ SortArrayContext sacontext;
+ ClauseContext *ccxt;
+ Const *con;
+ ScalarArrayOpExpr *saop = (ScalarArrayOpExpr *)expr;
+ arrayval = DatumGetArrayTypeP(((Const *) rightop)->constvalue);
+ /* We could cache this data, but not clear it's worth it */
+ get_typlenbyvalalign(ARR_ELEMTYPE(arrayval),
+ &sacontext.elmlen, &sacontext.elmbyval, &sacontext.elmalign);
+ sacontext.elmtype = ARR_ELEMTYPE(arrayval);
+ deconstruct_array(arrayval,
+ ARR_ELEMTYPE(arrayval),
+ sacontext.elmlen, sacontext.elmbyval, sacontext.elmalign,
+ &elem_values, &elem_nulls, &num_elems);
+
+ /*
+ * Compress out any null elements. We can ignore them since we assume
+ * all B-tree operators are strict.
+ */
+ num_nonnulls = 0;
+ for (j = 0; j < num_elems; j++)
+ {
+ if (!elem_nulls[j])
+ elem_values[num_nonnulls++] = elem_values[j];
+ }
+ if (num_nonnulls == 0)
+ return true;
+
+ /* If NULL is existed and with ALL flag, report it. */
+ if (!saop->useOr && num_nonnulls != num_elems)
+ return false;
+
+ /* Sort the non-null elements and eliminate any duplicates.
+ * If there is no support function will return 0 and then
+ * give up.
+ */
+ if ( 0 == (num_elems = sort_scalararray_elements(&sacontext, right_type, opfamily, collid,
+ false, elem_values, num_nonnulls)))
+ return true;
+
+ attinfo = get_varatt_info_internal(sec, ea, (Var *) leftop);
+ if (unlikely(strat == STRATEGY_NOT_EQUAL))
+ {
+ /* operator <> */
+ if (!saop->useOr)
+ {
+ for (j = 0; j < num_elems; j++)
+ {
+ if (!add_binary_op_clause_info(attinfo,
+ &elem_values[j],
+ 1,
+ idx,
+ strat,
+ opfuncid,
+ expr,
+ sec,
+ ccw))
+ return false;
+ ea->addOp = true;
+ }
+ }
+ else
+ {
+ /* Treat it as and-expr implicitly if there is only one element. */
+ if (num_elems != 1)
+ return true; /* Don't flatten, forget it. */
+ con = makeConst(left_type,
+ -1, /* typmod -1 is OK for all cases */
+ InvalidOid,
+ sacontext.elmlen,
+ elem_values[0],
+ false,
+ sacontext.elmbyval);
+ con->location = -1;
+ newclause = make_opclause(opid,
+ BOOLOID,
+ false,
+ (Expr*)leftop,
+ (Expr*)con,
+ InvalidOid,
+ collid);
+ if (!add_binary_op_clause_info(attinfo,
+ &elem_values[0],
+ 1,
+ idx,
+ strat,
+ opfuncid,
+ newclause,
+ sec,
+ ccw))
+ return false;
+ ea->addOp = true;
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = idx;
+ ccxt->cxt = newclause;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ else
+ {
+ switch (strat)
+ {
+ case BTLessStrategyNumber:
+ case BTLessEqualStrategyNumber:
+ if (!saop->useOr)
+ one_elem = &elem_values[0];
+ else
+ one_elem = &elem_values[num_elems-1];
+ break;
+ case BTEqualStrategyNumber:
+ /* If the number of elements is not one and postpone them,
+ * examine them later, because maybe there are multiple
+ * any-op ScalarArrayOpExpr (eg. x in(1,3) and x in(1,5)),
+ * see self_contradictory_saop comment.
+ */
+ if (num_elems != 1)
+ {
+ if (!saop->useOr)
+ return false; /* It's contradictory obviously. */
+ else
+ {
+ SAEqualDatElem *saopele;
+ if (attinfo->saop.elemnents == NIL)
+ {
+ attinfo->saop.opfuncid = opfuncid;
+ memcpy(&(attinfo->saop.sortcxt), &sacontext, sizeof(SortArrayContext));
+ }
+ saopele = palloc0(sizeof(SAEqualDatElem));
+ saopele->idx = idx;
+ saopele->expr = expr;
+ saopele->elem_nums = num_elems;
+ saopele->elem_values = elem_values;
+ attinfo->saop.elemnents = lappend(attinfo->saop.elemnents, saopele);
+ }
+ }
+ else
+ {
+ Assert(num_elems == 1);
+ one_elem = &elem_values[0];
+ }
+ break;
+ case BTGreaterEqualStrategyNumber:
+ case BTGreaterStrategyNumber:
+ if (!saop->useOr)
+ one_elem = &elem_values[num_elems-1];
+ else
+ one_elem = &elem_values[0];
+ break;
+ default:
+ elog(ERROR, "unrecognized StrategyNumber: %d",
+ strat);
+ break;
+ }
+ if (!(strat == BTEqualStrategyNumber && num_elems != 1))
+ {
+ con = makeConst(left_type,
+ -1, /* typmod -1 is OK for all cases */
+ InvalidOid,
+ sacontext.elmlen,
+ *one_elem,
+ false,
+ sacontext.elmbyval);
+ con->location = -1;
+ newclause = make_opclause(opid,
+ BOOLOID,
+ false,
+ (Expr*)leftop,
+ (Expr*)con,
+ InvalidOid,
+ collid);
+ if (!add_binary_op_clause_info(attinfo,
+ one_elem,
+ 1,
+ idx,
+ strat,
+ opfuncid,
+ newclause,
+ sec,
+ ccw))
+ return false;
+
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = idx;
+ ccxt->cxt = newclause;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ }
+ else
+ {
+ attinfo = get_varatt_info_internal(sec, ea, (Var *) leftop);
+ if (!add_binary_op_clause_info(attinfo,
+ &((Const *) rightop)->constvalue,
+ 1,
+ idx,
+ strat,
+ opfuncid,
+ expr,
+ sec,
+ ccw))
+ return false;
+ }
+ ea->addOp = true;
+
+ return true;
+}
+
+/*
+ * row_comparsion_to_nonconstructor
+ * Convert a row comparison to nonconstructor
+ *
+ * We have checked the expression in analysis phase, so
+ * there is no need to check whether it can satisfy the
+ * required of B-tree or not.
+ */
+static int row_comparsion_to_nonconstructor(Expr *rowexpr, Expr **transformed)
+{
+ Oid left_type, right_type;
+ Oid varcollid;
+ ListCell *l_left_expr,
+ *l_right_expr,
+ *l_opno,
+ *l_opfamily,
+ *l_inputcollid;
+ ListCell *lc;
+ List *rcelst, *arglst;
+ BoolExpr *orclause, *andclause;
+ RowCompElem *rce;
+ Expr *newclause;
+ RowCompareExpr *rcexpr;
+ FmgrInfo op_func;
+ bool result, lasteva;
+
+ *transformed = NULL;
+ rcexpr = (RowCompareExpr *) rowexpr;
+ Assert(list_length(rcexpr->opnos) != 1);
+ Assert(rcexpr->rctype != ROWCOMPARE_EQ && rcexpr->rctype != ROWCOMPARE_NE);
+
+ rcelst = NIL;
+ orclause = (BoolExpr*)make_orclause(NIL);
+ forfive(l_left_expr, rcexpr->largs,
+ l_right_expr, rcexpr->rargs,
+ l_opno, rcexpr->opnos,
+ l_opfamily, rcexpr->opfamilies,
+ l_inputcollid, rcexpr->inputcollids)
+ {
+ RegProcedure opfuncid;
+ int strat;
+ Oid eqid;
+ Expr *tmp_expr;
+ Expr *left_expr = (Expr *) lfirst(l_left_expr);
+ Expr *right_expr = (Expr *) lfirst(l_right_expr);
+ Oid opno = lfirst_oid(l_opno);
+ Oid opfamily = lfirst_oid(l_opfamily);
+ Oid inputcollid = lfirst_oid(l_inputcollid);
+
+ if ((IsA(left_expr, Const)
+ && ((Const*)left_expr)->constisnull)
+ || (IsA(right_expr, Const)
+ && ((Const*)right_expr)->constisnull))
+ {
+ if (list_cell_number(rcexpr->opnos, l_opno) == 0)
+ return CCT_CONSTF;
+ lasteva = false;
+ break; /* The rest of pairs are dominated by the NULL, ignore them. */
+ }
+ if (equal(left_expr, right_expr))
+ {
+ if (!contain_volatile_functions((Node*)left_expr))
+ {
+ if (!IsA(left_expr, Const))
+ {
+ NullTest *nulltest = makeNode(NullTest);
+ nulltest->arg = left_expr;
+ nulltest->nulltesttype = IS_NOT_NULL;
+ nulltest->argisrow = false;
+ nulltest->location = -1;
+ rce = palloc0(sizeof(RowCompElem));
+ rce->contact = (Expr*)nulltest;
+ rcelst = lappend(rcelst, rce);
+ }
+ /* IF the last operator is > or <, the last sub-expr must be failed. */
+ if (l_opno == list_last_cell(rcexpr->opnos))
+ {
+ strat = get_op_opfamily_strategy(opno, opfamily);
+ if (strat == BTLessStrategyNumber || strat == BTGreaterStrategyNumber)
+ lasteva = false;
+ else
+ lasteva = true;
+ if (lasteva)
+ {
+ arglst = NIL;
+ foreach(lc, rcelst)
+ {
+ arglst = lappend(arglst, ((RowCompElem*)lfirst(lc))->contact);
+ }
+ goto lab_built;
+ }
+ else
+ break;
+ }
+ continue;
+ }
+ else
+ {
+ /* Forget this rcexpr, cann't do anything more. */
+ *transformed = NULL;
+ return 0;
+ }
+ }
+
+ /* If Const Op Const and are not equal. It's need to evaluate
+ * the expt is positive or not.
+ */
+ if (IsA(left_expr, Const) && IsA(right_expr, Const))
+ {
+ opfuncid = get_opcode(opno);
+ fmgr_info(opfuncid, &op_func);
+ result = DatumGetBool(FunctionCall2Coll(&op_func,
+ inputcollid,
+ ((Const *) left_expr)->constvalue,
+ ((Const *) right_expr)->constvalue));
+ if (result)
+ {
+ if (list_cell_number(rcexpr->opnos, l_opno) == 0)
+ return CCT_CONSTT;
+ else
+ {
+ arglst = NIL;
+ foreach(lc, rcelst)
+ {
+ arglst = lappend(arglst, ((RowCompElem*)lfirst(lc))->contact);
+ }
+ if (list_length(arglst) == 1)
+ {
+ orclause->args = lappend(orclause->args, linitial(arglst));
+ }
+ else if (list_length(arglst) > 1)
+ {
+ andclause = (BoolExpr*) make_andclause(arglst);
+ orclause->args = lappend(orclause->args, andclause);
+ }
+ lasteva = true;
+ break;
+ }
+ }
+ else
+ {
+ if (list_cell_number(rcexpr->opnos, l_opno) == 0)
+ return CCT_CONSTF;
+ else
+ {
+ lasteva = false;
+ break;
+ }
+ }
+ }
+
+ /* Only accept Var op Const, others cause the rcexpr will be ignored. */
+ if (!(IsA(left_expr, Const) || IsA(right_expr, Const)))
+ {
+ *transformed = NULL;
+ return 0;
+ }
+
+ if (IsA(left_expr, Const))
+ {
+ opno = get_commutator(opno);
+ tmp_expr = left_expr;
+ left_expr = right_expr;
+ right_expr = tmp_expr;
+ }
+
+ tmp_expr = left_expr;
+ if (IsA(left_expr, RelabelType))
+ {
+ left_type = ((RelabelType *) left_expr)->resulttype;
+ varcollid = ((RelabelType *) left_expr)->resultcollid;
+ left_expr = ((RelabelType *) left_expr)->arg;
+ }
+
+ if (IsA(left_expr, Var))
+ {
+ left_type = ((Var *) left_expr)->vartype;
+ varcollid = ((Var *) left_expr)->varcollid;
+ }
+ else
+ {
+ *transformed = NULL;
+ return 0;
+ }
+ right_type = ((Const *) right_expr)->consttype;
+ if ( left_type != right_type
+ || inputcollid != varcollid) /* Don't spend much effort for other types, so skip it */
+
+ {
+ *transformed = NULL;
+ return 0;
+ }
+ /* Built sub expr. */
+ rce = palloc0(sizeof(RowCompElem));
+ rce->opid = opno;
+ eqid = get_opfamily_member(opfamily, left_type, left_type, BTEqualStrategyNumber);
+ newclause = make_opclause(eqid,
+ BOOLOID,
+ false,
+ tmp_expr,
+ right_expr,
+ InvalidOid,
+ inputcollid);
+ rce->contact = newclause;
+ strat = get_op_opfamily_strategy(opno, opfamily);
+ if (strat == BTLessEqualStrategyNumber)
+ rce->stricter_opid = get_opfamily_member(opfamily, left_type, left_type, BTLessStrategyNumber);
+ else if (strat == BTGreaterEqualStrategyNumber)
+ rce->stricter_opid = get_opfamily_member(opfamily, left_type, left_type, BTGreaterStrategyNumber);
+ else
+ rce->stricter_opid = opno;
+
+ if (l_opno == list_last_cell(rcexpr->opnos))
+ opno = rce->opid;
+ else
+ opno = rce->stricter_opid;
+ newclause = make_opclause(opno,
+ BOOLOID,
+ false,
+ tmp_expr,
+ right_expr,
+ InvalidOid,
+ inputcollid);
+
+ arglst = NIL;
+ foreach(lc, rcelst)
+ {
+ arglst = lappend(arglst, ((RowCompElem*)lfirst(lc))->contact);
+ }
+ arglst = lappend(arglst, newclause);
+ rcelst = lappend(rcelst, rce);
+
+lab_built:
+ if (list_length(arglst) == 1)
+ {
+ orclause->args = lappend(orclause->args, linitial(arglst));
+ }
+ else if (list_length(arglst) > 1)
+ {
+ andclause = (BoolExpr*)make_andclause(NIL);
+ andclause->args = arglst;
+ orclause->args = lappend(orclause->args, andclause);
+ }
+ }
+
+ /* If or-clause args is empty, that mean all of pairs of elements
+ * are const equal, so check the result of last sub-expr.
+ */
+ if (orclause->args == NIL)
+ {
+ if (lasteva)
+ return CCT_CONSTT;
+ else
+ return CCT_CONSTF;
+ }
+
+ if (list_length(orclause->args) == 1)
+ {
+ *transformed = linitial(orclause->args);
+ }
+ else
+ *transformed = (Expr*)orclause;
+#ifdef DEBUG_ROW_DECODE
+ elog(NOTICE, "ROW COMPARSION TO NONCONSTRUCTOR: %s", nodeToString(*transformed));
+#endif
+ return 0;
+}
+
+
+/*
+ * self_contradictory_recurse
+ * Extract particulares of implict-AND clauses list
+ * passed in and check whether it
+ * is self contradictory or not. During the time of invoking routine
+ * also do the following jobs pruning and replacing and deleting clause.
+ *
+ * Returned value:
+ * 0: it's Ok.
+ * 1: is self-contradictory
+ * Function expect a couple of caluse that mean these clauses are
+ * implict-AND caluses. Once a pair of clauses or a clause are
+ * self-contradictory report to the caller immediately.
+ * A or-clause is a bit tricky, it's need to check every arg, if
+ * sub-arg is self-contradictory abandon it. After recursion
+ * if all of them are self contradictory that mean the or-clause
+ * is self contradictory and report to the caller, else rebuilt the
+ * or-clause (eg. x > 2 and x < 1 or y >3 in the case x > 2 and x < 1
+ * will be eliminated, the output expression is y > 3 ).
+ * We can push other expressions these have the same semantic level
+ * with OR-expression down into the OR-expression
+ * (eg. (x > 2 and x < 1 or y > 3) and y < 2, y < 2 will be
+ * push down to sub-arg y > 3, this clause is self-contradictory).
+ */
+static int self_contradictory_recurse(List *node,
+ ExtractedAttris *ea,
+ ClauseCxtWapper *ccw,
+ SelfExamContext *sec)
+{
+ int ret = 0;
+ int i;
+ Expr *inter_node;
+ VarattInfo *att_elem;
+ ListCell *lc;
+ ClauseContext *clausecxt;
+ NullTest *nt_node;
+ Var *varnode;
+ ExtractedAttris ea_info;
+ ClauseCxtWapper orccw;
+
+ ea_info.addOp = false;
+ ea_info.min_attr = sec->attrsdc.min_attr;
+ ea_info.att_lst = NULL;
+ ea_info.upper_ea = NULL;
+ ea_info.array_vi = palloc0((sec->attrsdc.max_attr - sec->attrsdc.min_attr + 1) * sizeof(VarattInfo*));
+ orccw.clausecxt = NIL;
+ if (ea)
+ ea_info.upper_ea = ea;
+
+ if (IsA(node, List))
+ {
+ i = -1;
+ foreach(lc, (List*) node)
+ {
+ inter_node = lfirst(lc);
+ i++;
+ if (IsA(inter_node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) inter_node;
+ if (rinfo->pseudoconstant)
+ continue;
+ inter_node = rinfo->clause;
+ if (rinfo->fromec)
+ {
+ clausecxt = get_clause_cxt_internal(sec);
+ clausecxt->cct = CCT_DEL;
+ clausecxt->idx = i;
+ ccw->clausecxt = lappend(ccw->clausecxt, clausecxt);
+ }
+ }
+ if (IsA(inter_node, Var))
+ {
+ varnode = (Var*) inter_node;
+ if (varnode->vartype == BOOLOID)
+ {
+ att_elem = get_varatt_info_internal(sec, &ea_info, varnode);
+ att_elem->bool_test |= BT_MARK | BT_IS_TRUE;
+ ea_info.addOp = true;
+ }
+ }
+ else if (IsA(inter_node, BoolExpr))
+ {
+ /*
+ * It's need to postpone the or-expr, then the same
+ * level simple expr can be push down.
+ */
+ BoolExpr *boolnode;
+ boolnode = (BoolExpr*) inter_node;
+ if (boolnode->boolop == NOT_EXPR
+ && list_length(boolnode->args) == 1
+ && IsA(linitial(boolnode->args), Var))
+ {
+ varnode = (Var*) linitial(boolnode->args);
+ if (varnode->vartype == BOOLOID)
+ {
+ att_elem = get_varatt_info_internal(sec, &ea_info, varnode);
+ att_elem->bool_test |= BT_MARK | BT_IS_FALSE;
+ ea_info.addOp = true;
+ }
+ }
+ else
+ {
+ clausecxt = get_clause_cxt_internal(sec);
+ clausecxt->idx = i;
+ clausecxt->cxt = inter_node;
+ clausecxt->origin = inter_node;
+ orccw.clausecxt = lappend(orccw.clausecxt, clausecxt);
+ }
+ }
+ else if (IsA(inter_node, NullTest))
+ {
+ nt_node = (NullTest *)inter_node;
+ if (!nt_node->argisrow && nt_node->arg)
+ {
+ if(!deconstruct_nulltest_attr((Node *)nt_node->arg,
+ nt_node->nulltesttype, sec, &ea_info))
+ EXAMINE_ABORT(1);
+ ea_info.addOp = true;
+ }
+ }
+ else if (IsA(inter_node, BooleanTest))
+ {
+ Node *basenode;
+
+ BooleanTest *btest = (BooleanTest *) inter_node;
+ if (IsA(btest->arg, RelabelType))
+ basenode = (Node *) ((RelabelType *) btest->arg)->arg;
+ else
+ basenode = (Node *)btest->arg;
+ if (basenode && IsA(basenode, Var))
+ {
+ att_elem = get_varatt_info_internal(sec, &ea_info, (Var *) btest->arg);
+ switch (btest->booltesttype)
+ {
+ case IS_TRUE:
+ att_elem->bool_test |= BT_MARK | BT_IS_TRUE;
+ break;
+ case IS_NOT_TRUE:
+ att_elem->bool_test |= BT_MARK | BT_IS_NOT_TRUE;
+ break;
+ case IS_FALSE:
+ att_elem->bool_test |= BT_MARK | BT_IS_FALSE;
+ break;
+ case IS_NOT_FALSE:
+ att_elem->bool_test |= BT_MARK | BT_IS_NOT_FALSE;
+ break;
+ case IS_UNKNOWN:
+ att_elem->bool_test |= BT_MARK | BT_IS_UNKNOWN;
+ break;
+ case IS_NOT_UNKNOWN:
+ att_elem->bool_test |= BT_MARK | BT_IS_NOT_UNKNOWN;
+ break;
+ default:
+ elog(ERROR, "unrecognized booltesttype: %d",
+ (int) btest->booltesttype);
+ break;
+ }
+ ea_info.addOp = true;
+ }
+ }
+ else if (IsA(inter_node, OpExpr))
+ {
+ OpExpr *opexpr;
+ opexpr = (OpExpr *) inter_node;
+ if (list_length(opexpr->args) != 2)
+ continue;
+ if (!extract_binary_op_clause_info(opexpr,
+ linitial(opexpr->args),
+ lsecond(opexpr->args),
+ i,
+ opexpr->opno,
+ opexpr->inputcollid,
+ sec,
+ &ea_info,
+ ccw))
+ EXAMINE_ABORT(1);
+ }
+ else if (IsA(inter_node, ScalarArrayOpExpr))
+ {
+ ScalarArrayOpExpr *saop = (ScalarArrayOpExpr *) inter_node;
+ if (list_length(saop->args) != 2)
+ continue;
+ if (!extract_binary_op_clause_info(saop,
+ linitial(saop->args),
+ lsecond(saop->args),
+ i,
+ saop->opno,
+ saop->inputcollid,
+ sec,
+ &ea_info,
+ ccw))
+ EXAMINE_ABORT(1);
+ }
+ else if (IsA(inter_node, RowCompareExpr))
+ {
+ Expr *trans;
+ int result;
+ result = row_comparsion_to_nonconstructor(inter_node, &trans);
+ if (result == 0 && trans != NULL)
+ {
+ if (is_orclause(trans))
+ {
+ clausecxt = get_clause_cxt_internal(sec);
+ clausecxt->idx = i;
+ clausecxt->cxt = trans;
+ clausecxt->origin = inter_node;
+ orccw.clausecxt = lappend(orccw.clausecxt, clausecxt);
+ }
+ else if (is_andclause(trans))
+ {
+ /* Must be NullTest expr eg. row(a,b) >= row(a,b) */
+ ListCell *lnt;
+ foreach(lnt, ((BoolExpr*)trans)->args)
+ {
+ Assert(IsA(lfirst(lnt), NullTest));
+ nt_node = lfirst(lnt);
+ if(!deconstruct_nulltest_attr((Node *)nt_node->arg,
+ nt_node->nulltesttype,
+ sec,
+ &ea_info))
+ EXAMINE_ABORT(1);
+ }
+ ea_info.addOp = true;
+ }
+ else if (IsA(trans, OpExpr))
+ {
+ if (!extract_binary_op_clause_info(trans,
+ linitial(((OpExpr*)trans)->args),
+ lsecond(((OpExpr*)trans)->args),
+ i,
+ ((OpExpr*)trans)->opno,
+ ((OpExpr*)trans)->inputcollid,
+ sec,
+ &ea_info,
+ ccw))
+ EXAMINE_ABORT(1);
+ clausecxt = get_clause_cxt_internal(sec);
+ clausecxt->cct = CCT_REPLACE;
+ clausecxt->idx = i;
+ clausecxt->cxt = trans;
+ ccw->clausecxt = lappend(ccw->clausecxt, clausecxt);
+ }
+ else if (IsA(trans, NullTest))
+ {
+ nt_node = (NullTest *)trans;
+ if(!deconstruct_nulltest_attr((Node *)nt_node->arg,
+ nt_node->nulltesttype,
+ sec,
+ &ea_info))
+ EXAMINE_ABORT(1);
+ clausecxt = get_clause_cxt_internal(sec);
+ clausecxt->cct = CCT_REPLACE;
+ clausecxt->idx = i;
+ clausecxt->cxt = trans;
+ ccw->clausecxt = lappend(ccw->clausecxt, clausecxt);
+ }
+ }
+ else if (result == CCT_CONSTT)
+ {
+ /* (eg. row(1,2,3) >= row(1,2,3)) */
+ clausecxt = get_clause_cxt_internal(sec);
+ clausecxt->cct = CCT_CONSTT;
+ clausecxt->idx = i;
+ ccw->clausecxt = lappend(ccw->clausecxt, clausecxt);
+ }
+ else if (result == CCT_CONSTF)
+ EXAMINE_ABORT(1);
+ }
+ }
+ }
+ else
+ {
+ elog(ERROR, "unrecognized node type: %d",
+ (int) nodeTag(node));
+ }
+
+ /* All base particulars had been collected, it's time to examine. */
+ if(self_contradictory_check(sec, &ea_info, ccw))
+ EXAMINE_ABORT(1);
+
+ /* Examine the or-clause that was postponed here because every
+ * OpExpr at same semantic level can be push down ( eg. x > 10 and y > 12 and
+ * ( x < 9 or y < 11) ), this case can be transform to (x > 10 and y > 12 and
+ * x <9) or (x > 10 and y > 12 and y < 11).
+ */
+ foreach(lc, orccw.clausecxt)
+ {
+ ListCell *arg, *lc1;
+ List *arglst, *newraglst;
+ ClauseCxtWapper boolccw;
+ ClauseContext *ccxt;
+ BoolExpr *boolnode;
+ uint8 *delidx;
+ uint8 orchanged;
+ clausecxt = lfirst(lc);
+ boolnode = (BoolExpr *) clausecxt->cxt;
+
+ switch (boolnode->boolop)
+ {
+ case AND_EXPR:
+ delidx = palloc0((list_length(boolnode->args)+1) * sizeof(uint8));
+ delidx++;
+ boolccw.clausecxt = NIL;
+ ret = self_contradictory_recurse(boolnode->args, ea, &boolccw, sec);
+ if (ret > 0)
+ EXAMINE_ABORT(1);
+
+ foreach(lc1, boolccw.clausecxt)
+ {
+ ccxt = lfirst(lc1);
+ delidx[-1] = 0x01;
+ switch (ccxt->cct)
+ {
+ case CCT_DEL:
+ case CCT_CONSTT:
+ delidx[ccxt->idx] = 0x01;
+ break;
+ case CCT_REPLACE:
+ arg = list_nth_cell(boolnode->args, ccxt->idx);
+ /* It's need to be flatten if a and-expr, so flag it. */
+ if (is_andclause(ccxt->cxt))
+ delidx[ccxt->idx] = 0x02;
+ lfirst(arg) = ccxt->cxt;
+ break;
+ default:
+ break;
+ }
+ free_clause_cxt_internal(sec, ccxt);
+ }
+ /* OK, fix the args. */
+ if (delidx[-1])
+ {
+ i = 0;
+ arglst = NIL;
+ newraglst = NIL;
+ foreach(lc1, boolnode->args)
+ {
+ if (delidx[i] == 0x00)
+ arglst = lappend(arglst, lfirst(lc1));
+ else if (delidx[i] == 0x02)
+ newraglst = list_concat(newraglst, ((BoolExpr*)lfirst(lc1))->args);
+ i++;
+ }
+ boolnode->args = list_concat(arglst,newraglst);
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->idx = clausecxt->idx;
+ /* If the reset of args is empty, that mean the and-expr
+ * is const true, because only weaker restrictive expr will be
+ * delete and the args must be not empty.
+ */
+ if (boolnode->args == NIL)
+ ccxt->cct = CCT_CONSTT;
+ else
+ {
+ ccxt->cct = CCT_REPLACE;
+ if (list_length(boolnode->args) == 1)
+ ccxt->cxt = linitial(boolnode->args);
+ else
+ ccxt->cxt = boolnode;
+ }
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ break;
+ case OR_EXPR:
+ newraglst = NIL;
+ orchanged = 0x0;
+ foreach(arg, boolnode->args)
+ {
+ boolccw.clausecxt = NIL;
+ arglst = list_make1(lfirst(arg));
+ if (self_contradictory_recurse(arglst, &ea_info, &boolccw, sec))
+ {
+ orchanged = 0x01;
+ ret++;
+ }
+ else
+ {
+ Assert(list_length(boolccw.clausecxt) <= 1);
+ if (list_length(boolccw.clausecxt) == 1)
+ {
+ orchanged = 0x01;
+ ccxt = linitial(boolccw.clausecxt);
+ Assert(ccxt->idx == 0);
+ if (ccxt->cct == CCT_REPLACE)
+ lfirst(arg) = ccxt->cxt;
+ else if (ccxt->cct == CCT_CONSTT)
+ {
+ /*
+ * Once found a const true sub-arg, skip checking any others,
+ * the or-expr must be const true.
+ */
+ orchanged = 0x02;
+ free_clause_cxt_internal(sec, ccxt);
+ list_free(arglst);
+ break;
+ }
+ free_clause_cxt_internal(sec, ccxt);
+ }
+ if (is_orclause(lfirst(arg)))
+ newraglst = list_concat(newraglst, ((const BoolExpr *) lfirst(arg))->args);
+ else
+ newraglst = lappend(newraglst, lfirst(arg));
+ }
+ list_free(arglst);
+ }
+ if (ret == list_length(boolnode->args))
+ EXAMINE_ABORT(1);
+#ifndef DEBUG_ROW_DECODE
+ else if (orchanged == 0x01)
+ {
+ /*
+ * RowCompareExpr has been coverted to nonconstructor (OR-expression),
+ * if the number of the or-expr's args is more than
+ * one, it do not has benifit to subsequent planning, so
+ * do not do anything and keep itself.
+ */
+
+ if ((IsA(clausecxt->origin, RowCompareExpr)
+ && list_length(newraglst) != 1))
+ ;
+ else
+#else
+ else if (orchanged == 0x01 || (IsA(clausecxt->origin, RowCompareExpr) && orchanged != 0x02))
+ {
+#endif
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_REPLACE;
+ ccxt->idx = clausecxt->idx;
+ if (list_length(newraglst) == 1)
+ ccxt->cxt = linitial(newraglst);
+ else
+ {
+ boolnode->args = newraglst;
+ ccxt->cxt = boolnode;
+ }
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ }
+ else if (orchanged == 0x02)
+ {
+ ccxt = get_clause_cxt_internal(sec);
+ ccxt->cct = CCT_CONSTT;
+ ccxt->idx = clausecxt->idx;
+ ccw->clausecxt = lappend(ccw->clausecxt, ccxt);
+ }
+ /* If or-expr is not changed, do nothing more. */
+ ret = 0;
+ break;
+ case NOT_EXPR:
+ break;
+ default:
+ elog(ERROR, "unrecognized boolop: %d", (int) boolnode->boolop);
+ break;
+ }
+ free_clause_cxt_internal(sec, clausecxt);
+ }
+
+finished:
+ if (ea_info.att_lst != NULL && ea_info.att_lst == ea_info.last)
+ {
+ free_attinfo_cxt_internal(sec, ea_info.att_lst);
+ }
+ else if (ea_info.att_lst != NULL)
+ {
+ ea_info.last->next = sec->vifree;
+ sec->vifree = ea_info.att_lst;
+ }
+ pfree(ea_info.array_vi);
+ return ret;
+}
+
+/*
+ * examine_self_contradictory_rels
+ *
+ * Examine the base rel's baserestrictinfo to confirm that it's a
+ * self-contradictory rel or not.
+ * Mark the rel as a dummy rel if it's a self-contradictory rel,
+ * this is a huge win for subsequent planning.
+ * Note: deduct will be set true if caller need to process EC.
+ */
+static void examine_self_contradictory_rels(PlannerInfo *root, RelOptInfo *rel, bool deduct)
+{
+ int n;
+ RestrictInfo *restrictinfo;
+ ClauseCxtWapper ccwapper;
+ SelfExamContext sec;
+
+ MemSet(&sec, 0, sizeof(SelfExamContext));
+ /*
+ * Initialize sopcinfo with opclass info. Extract all bt-am opclass
+ * particulars and sort by opcintype.
+ */
+ if (sec.sopcinfo.n_members == 0)
+ {
+ if (!default_opclassinfo_for_am(BTREE_AM_OID, extract_opc_info, &sec.sopcinfo))
+ elog(ERROR, "non default opclass records for AM %d", (int) BTREE_AM_OID);
+ qsort(sec.sopcinfo.opcies, sec.sopcinfo.n_members, sizeof(OpcInfoElement*), opc_element_cmp);
+ }
+
+ /* For every base rel reuse the AttrsDetailsCache memory. */
+ if (!sec.attrsdc.atop_array)
+ {
+ sec.attrsdc.atop_array = palloc0((rel->max_attr - rel->min_attr + 1) * sizeof(AttrOpFamily*));
+ sec.attrsdc.atop_set = palloc0(sizeof(AttrOpFamilySet));
+ }
+ else
+ {
+ AttrOpFamilySet *atfs, *atfs_tmp;
+ atfs = sec.attrsdc.atop_set->next;
+ n = rel->max_attr - rel->min_attr + 1;
+ /* If the number of current rel attri greater than previous rel and reset it. */
+ if (n > sec.attrsdc.max_attr - sec.attrsdc.min_attr + 1)
+ {
+ pfree(sec.attrsdc.atop_array);
+ sec.attrsdc.atop_array = palloc0(n * sizeof(AttrOpFamily*));
+ }
+ else
+ MemSet(sec.attrsdc.atop_array, 0, n * sizeof(AttrOpFamily*));
+ while(atfs)
+ {
+ atfs_tmp = atfs->next;
+ pfree(atfs);
+ atfs = atfs_tmp;
+ }
+ sec.attrsdc.len_set = 0;
+ }
+ sec.attrsdc.min_attr = rel->min_attr;
+ sec.attrsdc.max_attr = rel->max_attr;
+
+ ccwapper.clausecxt = NIL;
+ if (!self_contradictory_recurse(rel->baserestrictinfo, NULL, &ccwapper, &sec))
+ {
+ /* Check any pruned-clause and rebuilt it. */
+ List *arglst;
+ ListCell *br, *lc;
+ ListCell *arg;
+ BoolExpr *expression;
+ RestrictInfo *or_rinfo;
+ ClauseContext *ccxt;
+ bool *delidx = NULL;
+ if (list_length(ccwapper.clausecxt) > 0)
+ {
+ delidx = palloc0((list_length(rel->baserestrictinfo) + 1) * sizeof(bool));
+ delidx++;
+ }
+ foreach(lc, ccwapper.clausecxt)
+ {
+ ccxt = lfirst(lc);
+ br = list_nth_cell(rel->baserestrictinfo, ccxt->idx);
+ restrictinfo = lfirst(br);
+ switch (ccxt->cct)
+ {
+ case CCT_DEL:
+ /* Be carefull if it's keeped from EC,
+ * count baseri_nums because it will be delete below
+ * and can't count it in phase1.
+ */
+ if (deduct && restrictinfo->fromec)
+ rel->baseri_nums++;
+ delidx[-1] = delidx[ccxt->idx] = true;
+ break;
+ case CCT_CONSTT:
+ delidx[-1] = delidx[ccxt->idx] = true;
+ break;
+ case CCT_REPLACE:
+ Assert(!IsA(ccxt->cxt, List));
+ arglst = NIL;
+ if (is_andclause(ccxt->cxt))
+ {
+ expression = ccxt->cxt;
+ arglst = list_concat(arglst, expression->args);
+ }
+ else
+ arglst = lappend(arglst, ccxt->cxt);
+ delidx[-1] = delidx[ccxt->idx] = true;
+ foreach(arg, arglst)
+ {
+ or_rinfo = make_restrictinfo(root,
+ (Expr*) lfirst(arg),
+ restrictinfo->is_pushed_down,
+ restrictinfo->has_clone,
+ restrictinfo->is_clone,
+ restrictinfo->pseudoconstant,
+ restrictinfo->security_level,
+ NULL,
+ restrictinfo->incompatible_relids,
+ restrictinfo->outer_relids);
+ if (deduct)
+ {
+ check_mergejoinable(or_rinfo);
+ if (or_rinfo->mergeopfamilies)
+ {
+ if (restrictinfo->allow_equivalence)
+ {
+ if (process_equivalence(root, &or_rinfo, restrictinfo->jdomain))
+ {
+ rel->baseri_nums++;
+ continue;
+ }
+ if (or_rinfo->mergeopfamilies)
+ initialize_mergeclause_eclasses(root, or_rinfo);
+ }
+ else
+ initialize_mergeclause_eclasses(root, or_rinfo);
+ }
+ }
+ rel->baserestrictinfo = lappend(rel->baserestrictinfo, or_rinfo);
+ }
+ break;
+ default:
+ break;
+ }
+ free_clause_cxt_internal(&sec, ccxt);
+ }
+ if (delidx && delidx[-1])
+ {
+ List *rlist = NIL;
+ int i = 0;
+ foreach(lc, rel->baserestrictinfo)
+ {
+ if (!delidx[i++])
+ rlist = lappend(rlist, lfirst(lc));
+ }
+ rel->baserestrictinfo = rlist;
+ }
+ }
+ else
+ mark_dummy_rel(rel);
+}
+
+/*
+ * examine_self_contradictory_rels_phase1
+ *
+ * Some clause maybe be changed and the reult may match the
+ * EC mechanism, so it's need to add it into the EC.
+ */
+void examine_self_contradictory_rels_phase1(PlannerInfo *root)
+{
+ int rti;
+ RestrictInfo *rinfo;
+
+ for (rti = 1; rti < root->simple_rel_array_size; rti++)
+ {
+ RelOptInfo *rel = root->simple_rel_array[rti];
+ /* There may be empty slots corresponding to non-baserel RTEs */
+ if (rel == NULL)
+ continue;
+ /* Ignore any "otherrels". */
+ if (rel->reloptkind != RELOPT_BASEREL)
+ continue;
+ if (rel->baserestrictinfo == NIL)
+ continue;
+ if (list_length(rel->baserestrictinfo) == 1)
+ {
+ rinfo = linitial(rel->baserestrictinfo);
+ if (rinfo->fromec)
+ {
+ rel->baseri_nums = 1;
+ rel->baserestrictinfo = NIL;
+ continue;
+ }
+ }
+ examine_self_contradictory_rels(root, rel, true);
+ if (!is_dummy_rel(rel))
+ rel->baseri_nums += list_length(rel->baserestrictinfo);
+ }
+}
+
+/*
+ * examine_self_contradictory_rels_phase1
+ *
+ * Only care about rel's baserestrictinfo that had added a EC clause.
+ */
+void examine_self_contradictory_rels_phase2(PlannerInfo *root)
+{
+ int rti;
+ for (rti = 1; rti < root->simple_rel_array_size; rti++)
+ {
+ RelOptInfo *rel = root->simple_rel_array[rti];
+ /* There may be empty slots corresponding to non-baserel RTEs */
+ if (rel == NULL)
+ continue;
+ /* Ignore any "otherrels". */
+ if (rel->reloptkind != RELOPT_BASEREL)
+ continue;
+ if (rel->baserestrictinfo == NIL)
+ continue;
+ if (list_length(rel->baserestrictinfo) == 1)
+ continue;
+ if (is_dummy_rel(rel))
+ continue;
+ if (rel->baseri_nums != list_length(rel->baserestrictinfo))
+ examine_self_contradictory_rels(root, rel, false);
+ }
+}
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index 903c397d40..72445ba859 100644
--- a/src/backend/optimizer/plan/initsplan.c
+++ b/src/backend/optimizer/plan/initsplan.c
@@ -127,7 +127,7 @@ static void distribute_qual_to_rels(PlannerInfo *root, Node *clause,
List **postponed_oj_qual_list);
static bool check_redundant_nullability_qual(PlannerInfo *root, Node *clause);
static Relids get_join_domain_min_rels(PlannerInfo *root, Relids domain_relids);
-static void check_mergejoinable(RestrictInfo *restrictinfo);
+void check_mergejoinable(RestrictInfo *restrictinfo);
static void check_hashjoinable(RestrictInfo *restrictinfo);
static void check_memoizable(RestrictInfo *restrictinfo);
@@ -2534,7 +2534,8 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause,
relids,
incompatible_relids,
outerjoin_nonnullable);
-
+ restrictinfo->jdomain = jtitem->jdomain;
+ restrictinfo->allow_equivalence = allow_equivalence;
/*
* If it's a join clause, add vars used in the clause to targetlists of
* their relations, so that they will be emitted by the plan nodes that
@@ -2616,7 +2617,16 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause,
if (maybe_equivalence)
{
if (process_equivalence(root, &restrictinfo, jtitem->jdomain))
+ {
+ /* Distribute to base rel. */
+ int relid;
+ if (bms_get_singleton_member(restrictinfo->required_relids, &relid))
+ {
+ restrictinfo->fromec = true;
+ distribute_restrictinfo_to_rels(root, restrictinfo);
+ }
return;
+ }
/* EC rejected it, so set left_ec/right_ec the hard way ... */
if (restrictinfo->mergeopfamilies) /* EC might have changed this */
initialize_mergeclause_eclasses(root, restrictinfo);
@@ -3519,7 +3529,7 @@ match_foreign_keys_to_quals(PlannerInfo *root)
* the operator is a mergejoinable operator. The arguments can be
* anything --- as long as there are no volatile functions in them.
*/
-static void
+void
check_mergejoinable(RestrictInfo *restrictinfo)
{
Expr *clause = restrictinfo->clause;
diff --git a/src/backend/optimizer/plan/meson.build b/src/backend/optimizer/plan/meson.build
index c3b191cdf8..2f8c32fe5d 100644
--- a/src/backend/optimizer/plan/meson.build
+++ b/src/backend/optimizer/plan/meson.build
@@ -9,4 +9,5 @@ backend_sources += files(
'planner.c',
'setrefs.c',
'subselect.c',
+ 'contradictory.c',
)
diff --git a/src/backend/optimizer/plan/planmain.c b/src/backend/optimizer/plan/planmain.c
index e17d31a5c3..34cc104f1b 100644
--- a/src/backend/optimizer/plan/planmain.c
+++ b/src/backend/optimizer/plan/planmain.c
@@ -187,6 +187,8 @@ query_planner(PlannerInfo *root,
joinlist = deconstruct_jointree(root);
+ examine_self_contradictory_rels_phase1(root);
+
/*
* Reconsider any postponed outer-join quals now that we have built up
* equivalence classes. (This could result in further additions or
@@ -201,6 +203,8 @@ query_planner(PlannerInfo *root,
*/
generate_base_implied_equalities(root);
+ examine_self_contradictory_rels_phase2(root);
+
/*
* We have completed merging equivalence sets, so it's now possible to
* generate pathkeys in canonical form; so compute query_pathkeys and
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index a85dc0d891..bdebae5dbb 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1181,6 +1181,45 @@ get_language_name(Oid langoid, bool missing_ok)
/* ---------- OPCLASS CACHE ---------- */
+/*
+ * default_opclassinfo_for_am
+ *
+ * Extract all records these opcdefault is true with specified AM oid.
+ */
+bool
+default_opclassinfo_for_am(Oid amoid, SortOpclassCallback callback, void *infocxt)
+{
+ bool result = false;
+ CatCList *opclist;
+ int i;
+
+ /*
+ * Search through all the specified AM's opclass details to
+ * grab records which opcintype is true.
+ */
+ opclist = SearchSysCacheList1(CLAAMNAMENSP, ObjectIdGetDatum(amoid));
+
+ for (i = 0; i < opclist->n_members; i++)
+ {
+ HeapTuple classtup = &opclist->members[i]->tuple;
+ Form_pg_opclass classform = (Form_pg_opclass) GETSTRUCT(classtup);
+
+ if (classform->opcdefault)
+ {
+ callback(opclist->n_members,
+ classform->oid,
+ classform->opcfamily,
+ classform->opcintype,
+ infocxt);
+ }
+ }
+ if (opclist->n_members > 0)
+ result = true;
+ ReleaseCatCacheList(opclist);
+
+ return result;
+}
+
/*
* get_opclass_family
*
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index add0f9e45f..36d7e2bc21 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -982,6 +982,7 @@ typedef struct RelOptInfo
* used by various scans and joins:
*/
/* RestrictInfo structures (if base rel) */
+ uint32 baseri_nums;
List *baserestrictinfo;
/* cost of evaluating the above */
QualCost baserestrictcost;
@@ -2573,6 +2574,12 @@ typedef struct RestrictInfo
/* the represented clause of WHERE or JOIN */
Expr *clause;
+ JoinDomain *jdomain pg_node_attr(copy_as_scalar, equal_ignore, read_write_ignore);
+
+ bool fromec;
+
+ bool allow_equivalence;
+
/* true if clause was pushed down in level */
bool is_pushed_down;
diff --git a/src/include/optimizer/planmain.h b/src/include/optimizer/planmain.h
index 93137261e4..943e279bd8 100644
--- a/src/include/optimizer/planmain.h
+++ b/src/include/optimizer/planmain.h
@@ -66,6 +66,8 @@ extern Limit *make_limit(Plan *lefttree, Node *limitOffset, Node *limitCount,
*/
extern PGDLLIMPORT int from_collapse_limit;
extern PGDLLIMPORT int join_collapse_limit;
+extern void examine_self_contradictory_rels_phase1(PlannerInfo *root);
+extern void examine_self_contradictory_rels_phase2(PlannerInfo *root);
extern void add_base_rels_to_query(PlannerInfo *root, Node *jtnode);
extern void add_other_rels_to_query(PlannerInfo *root);
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index 20446f6f83..17faf447ef 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -61,6 +61,8 @@ typedef struct AttStatsSlot
void *numbers_arr; /* palloc'd numbers array, if any */
} AttStatsSlot;
+typedef void (*SortOpclassCallback) (int nums, Oid opcid, Oid opcfamily, Oid opcintype, void *infocxt);
+
/* Hook for plugins to get control in get_attavgwidth() */
typedef int32 (*get_attavgwidth_hook_type) (Oid relid, AttrNumber attnum);
extern PGDLLIMPORT get_attavgwidth_hook_type get_attavgwidth_hook;
@@ -103,6 +105,7 @@ extern Oid get_constraint_index(Oid conoid);
extern char get_constraint_type(Oid conoid);
extern char *get_language_name(Oid langoid, bool missing_ok);
+extern bool default_opclassinfo_for_am(Oid amoid, SortOpclassCallback callback, void *infocxt);
extern Oid get_opclass_family(Oid opclass);
extern Oid get_opclass_input_type(Oid opclass);
extern bool get_opclass_opfamily_and_input_type(Oid opclass,
--
2.43.0.windows.1
[application/octet-stream] regresssion_patch.diff (151.0K, ../../TYCPR01MB6093E2A06F3F8CF8DF17B5CE852E2@TYCPR01MB6093.jpnprd01.prod.outlook.com/4-regresssion_patch.diff)
download | inline diff:
From 065b6fca4a1457b1175fceb64de153716f671376 Mon Sep 17 00:00:00 2001
From: wq <[email protected]>
Date: Mon, 25 Nov 2024 16:32:59 +0800
Subject: [PATCH] Regression test of self contradictory examining on rel's
baserestrictinfo
---
src/test/regress/expected/create_index.out | 46 +-
src/test/regress/expected/equivclass.out | 15 +-
src/test/regress/expected/horology.out | 6 +-
src/test/regress/expected/inherit.out | 6 +-
src/test/regress/expected/join.out | 24 +-
src/test/regress/expected/partition_prune.out | 76 +-
src/test/regress/expected/rowsecurity.out | 18 +-
.../regress/expected/self_contradictory.out | 2607 +++++++++++++++++
src/test/regress/expected/stats_ext.out | 34 +-
src/test/regress/expected/updatable_views.out | 50 +-
src/test/regress/parallel_schedule | 3 +
src/test/regress/sql/self_contradictory.sql | 1481 ++++++++++
12 files changed, 4228 insertions(+), 138 deletions(-)
create mode 100644 src/test/regress/expected/self_contradictory.out
create mode 100644 src/test/regress/sql/self_contradictory.sql
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 1b0a5f0e9e..a0e460ef40 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2275,10 +2275,10 @@ ORDER BY thousand DESC, tenthous DESC;
--
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = ANY('{7, 8, 9}');
- QUERY PLAN
-----------------------------------------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------
Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 = ANY ('{1,42,7}'::integer[])) AND (unique1 = ANY ('{7,8,9}'::integer[])))
+ Index Cond: (unique1 = 7)
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = ANY('{7, 8, 9}');
@@ -2302,10 +2302,10 @@ SELECT unique1 FROM tenk1 WHERE unique1 = ANY('{7, 14, 22}') and unique1 = ANY('
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = 1;
- QUERY PLAN
----------------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------
Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 = ANY ('{1,42,7}'::integer[])) AND (unique1 = 1))
+ Index Cond: (unique1 = 1)
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = 1;
@@ -2316,10 +2316,10 @@ SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = 1;
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = 12345;
- QUERY PLAN
--------------------------------------------------------------------------------
- Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 = ANY ('{1,42,7}'::integer[])) AND (unique1 = 12345))
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = 12345;
@@ -2329,10 +2329,10 @@ SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 = 12345;
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 >= 42;
- QUERY PLAN
------------------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------------
Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 = ANY ('{1,42,7}'::integer[])) AND (unique1 >= 42))
+ Index Cond: ((unique1 >= 42) AND (unique1 = 42))
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 >= 42;
@@ -2343,10 +2343,10 @@ SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 >= 42;
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 > 42;
- QUERY PLAN
-----------------------------------------------------------------------------
- Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 = ANY ('{1,42,7}'::integer[])) AND (unique1 > 42))
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 > 42;
@@ -2356,10 +2356,10 @@ SELECT unique1 FROM tenk1 WHERE unique1 IN (1, 42, 7) and unique1 > 42;
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 > 9996 and unique1 >= 9999;
- QUERY PLAN
---------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------
Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 > 9996) AND (unique1 >= 9999))
+ Index Cond: (unique1 >= 9999)
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 > 9996 and unique1 >= 9999;
@@ -2370,10 +2370,10 @@ SELECT unique1 FROM tenk1 WHERE unique1 > 9996 and unique1 >= 9999;
explain (costs off)
SELECT unique1 FROM tenk1 WHERE unique1 < 3 and unique1 <= 3;
- QUERY PLAN
---------------------------------------------------
+ QUERY PLAN
+----------------------------------------------
Index Only Scan using tenk1_unique1 on tenk1
- Index Cond: ((unique1 < 3) AND (unique1 <= 3))
+ Index Cond: (unique1 < 3)
(2 rows)
SELECT unique1 FROM tenk1 WHERE unique1 < 3 and unique1 <= 3;
diff --git a/src/test/regress/expected/equivclass.out b/src/test/regress/expected/equivclass.out
index 5622750500..970186e5c8 100644
--- a/src/test/regress/expected/equivclass.out
+++ b/src/test/regress/expected/equivclass.out
@@ -143,12 +143,12 @@ explain (costs off)
explain (costs off)
select * from ec1, ec2 where ff = x1 and ff = '42'::int8;
- QUERY PLAN
--------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Join Filter: (ec1.ff = ec2.x1)
-> Index Scan using ec1_pkey on ec1
- Index Cond: ((ff = '42'::bigint) AND (ff = '42'::bigint))
+ Index Cond: (ff = '42'::bigint)
-> Seq Scan on ec2
(5 rows)
@@ -233,13 +233,12 @@ explain (costs off)
union all
select ff + 4 as x from ec1) as ss1
where ss1.x = ec1.f1 and ec1.ff = 42::int8 and ec1.ff = ec1.f1;
- QUERY PLAN
--------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------
Nested Loop
Join Filter: ((((ec1_1.ff + 2) + 1)) = ec1.f1)
-> Index Scan using ec1_pkey on ec1
- Index Cond: ((ff = '42'::bigint) AND (ff = '42'::bigint))
- Filter: (ff = f1)
+ Index Cond: (ff = '42'::bigint)
-> Append
-> Index Scan using ec1_expr2 on ec1 ec1_1
Index Cond: (((ff + 2) + 1) = '42'::bigint)
@@ -247,7 +246,7 @@ explain (costs off)
Index Cond: (((ff + 3) + 1) = '42'::bigint)
-> Index Scan using ec1_expr4 on ec1 ec1_3
Index Cond: ((ff + 4) = '42'::bigint)
-(12 rows)
+(11 rows)
explain (costs off)
select * from ec1,
diff --git a/src/test/regress/expected/horology.out b/src/test/regress/expected/horology.out
index 6d7dd5c988..4e075d6cdf 100644
--- a/src/test/regress/expected/horology.out
+++ b/src/test/regress/expected/horology.out
@@ -2541,11 +2541,11 @@ select count(*) from date_tbl
explain (costs off)
select count(*) from date_tbl
where f1 between symmetric '1997-01-01' and '1998-01-01';
- QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------------------------
Aggregate
-> Seq Scan on date_tbl
- Filter: (((f1 >= '01-01-1997'::date) AND (f1 <= '01-01-1998'::date)) OR ((f1 >= '01-01-1998'::date) AND (f1 <= '01-01-1997'::date)))
+ Filter: ((f1 >= '01-01-1997'::date) AND (f1 <= '01-01-1998'::date))
(3 rows)
select count(*) from date_tbl
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
index bb81f6d2b4..c8ad0af7a4 100644
--- a/src/test/regress/expected/inherit.out
+++ b/src/test/regress/expected/inherit.out
@@ -2766,10 +2766,10 @@ explain (costs off) select * from list_parted where a in ('ab', 'cd', 'ef');
(5 rows)
explain (costs off) select * from list_parted where a = 'ab' or a in (null, 'cd');
- QUERY PLAN
----------------------------------------------------------------------------------
+ QUERY PLAN
+----------------------------------------------------------
Seq Scan on part_ab_cd list_parted
- Filter: (((a)::text = 'ab'::text) OR ((a)::text = ANY ('{NULL,cd}'::text[])))
+ Filter: (((a)::text = 'ab'::text) OR (a = 'cd'::text))
(2 rows)
explain (costs off) select * from list_parted where a = 'ab';
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index ebf2e3f851..a8115414bf 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -8001,15 +8001,17 @@ where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1;
explain (costs off) select * from j1
inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2
where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1 and j2.id1 = any (array[1]);
- QUERY PLAN
-----------------------------------------------------
+ QUERY PLAN
+-------------------------------------------
Merge Join
- Merge Cond: (j1.id1 = j2.id1)
- Join Filter: (j2.id2 = j1.id2)
- -> Index Scan using j1_id1_idx on j1
- -> Index Scan using j2_id1_idx on j2
- Index Cond: (id1 = ANY ('{1}'::integer[]))
-(6 rows)
+ Merge Cond: (j1.id2 = j2.id2)
+ -> Index Only Scan using j1_pkey on j1
+ Index Cond: (id1 = 1)
+ Filter: ((id1 % 1000) = 1)
+ -> Index Only Scan using j2_pkey on j2
+ Index Cond: (id1 = 1)
+ Filter: ((id1 % 1000) = 1)
+(8 rows)
select * from j1
inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2
@@ -8024,14 +8026,14 @@ where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1 and j2.id1 = any (array[1]);
explain (costs off) select * from j1
inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2
where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1 and j2.id1 >= any (array[1,5]);
- QUERY PLAN
--------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------
Merge Join
Merge Cond: (j1.id1 = j2.id1)
Join Filter: (j2.id2 = j1.id2)
-> Index Scan using j1_id1_idx on j1
-> Index Scan using j2_id1_idx on j2
- Index Cond: (id1 >= ANY ('{1,5}'::integer[]))
+ Index Cond: (id1 >= 1)
(6 rows)
select * from j1
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 7a03b4e360..5debaf4ca0 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -632,10 +632,10 @@ explain (costs off) select * from rlp3 where a = 20; /* empty */
-- redundant clauses are eliminated
explain (costs off) select * from rlp where a > 1 and a = 10; /* only default */
- QUERY PLAN
-----------------------------------
+ QUERY PLAN
+--------------------------------
Seq Scan on rlp_default_10 rlp
- Filter: ((a > 1) AND (a = 10))
+ Filter: (a = 10)
(2 rows)
explain (costs off) select * from rlp where a > 1 and a >=15; /* rlp3 onwards, including default */
@@ -643,27 +643,27 @@ explain (costs off) select * from rlp where a > 1 and a >=15; /* rlp3 onwards, i
----------------------------------------------
Append
-> Seq Scan on rlp3abcd rlp_1
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp3efgh rlp_2
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp3nullxy rlp_3
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp3_default rlp_4
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp4_1 rlp_5
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp4_2 rlp_6
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp4_default rlp_7
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp5_1 rlp_8
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp5_default rlp_9
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp_default_30 rlp_10
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
-> Seq Scan on rlp_default_default rlp_11
- Filter: ((a > 1) AND (a >= 15))
+ Filter: (a >= 15)
(23 rows)
explain (costs off) select * from rlp where a = 1 and a = 3; /* empty */
@@ -674,20 +674,18 @@ explain (costs off) select * from rlp where a = 1 and a = 3; /* empty */
(2 rows)
explain (costs off) select * from rlp where (a = 1 and a = 3) or (a > 1 and a = 15);
- QUERY PLAN
--------------------------------------------------------------------
+ QUERY PLAN
+--------------------------------------
Append
- -> Seq Scan on rlp2 rlp_1
- Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
- -> Seq Scan on rlp3abcd rlp_2
- Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
- -> Seq Scan on rlp3efgh rlp_3
- Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
- -> Seq Scan on rlp3nullxy rlp_4
- Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
- -> Seq Scan on rlp3_default rlp_5
- Filter: (((a = 1) AND (a = 3)) OR ((a > 1) AND (a = 15)))
-(11 rows)
+ -> Seq Scan on rlp3abcd rlp_1
+ Filter: (a = 15)
+ -> Seq Scan on rlp3efgh rlp_2
+ Filter: (a = 15)
+ -> Seq Scan on rlp3nullxy rlp_3
+ Filter: (a = 15)
+ -> Seq Scan on rlp3_default rlp_4
+ Filter: (a = 15)
+(9 rows)
-- multi-column keys
create table mc3p (a int, b int, c int) partition by range (a, abs(b), c);
@@ -1561,10 +1559,10 @@ explain (costs off) select * from coercepart where a = any ('{ab,bc}');
(5 rows)
explain (costs off) select * from coercepart where a = any ('{ab,null}');
- QUERY PLAN
----------------------------------------------------
+ QUERY PLAN
+--------------------------------------
Seq Scan on coercepart_ab coercepart
- Filter: ((a)::text = ANY ('{ab,NULL}'::text[]))
+ Filter: (a = 'ab'::text)
(2 rows)
explain (costs off) select * from coercepart where a = any (null::text[]);
@@ -1575,10 +1573,10 @@ explain (costs off) select * from coercepart where a = any (null::text[]);
(2 rows)
explain (costs off) select * from coercepart where a = all ('{ab}');
- QUERY PLAN
-----------------------------------------------
+ QUERY PLAN
+--------------------------------------
Seq Scan on coercepart_ab coercepart
- Filter: ((a)::text = ALL ('{ab}'::text[]))
+ Filter: (a = 'ab'::text)
(2 rows)
explain (costs off) select * from coercepart where a = all ('{ab,bc}');
@@ -4174,20 +4172,20 @@ create table rp_prefix_test3_p2 partition of rp_prefix_test3 for values from (2,
-- Test that get_steps_using_prefix() handles a prefix that contains multiple
-- clauses for the partition key b (ie, b >= 1 and b >= 2)
explain (costs off) select * from rp_prefix_test3 where a >= 1 and b >= 1 and b >= 2 and c >= 2 and d >= 0;
- QUERY PLAN
---------------------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------
Seq Scan on rp_prefix_test3_p2 rp_prefix_test3
- Filter: ((a >= 1) AND (b >= 1) AND (b >= 2) AND (c >= 2) AND (d >= 0))
+ Filter: ((a >= 1) AND (b >= 2) AND (c >= 2) AND (d >= 0))
(2 rows)
-- Test that get_steps_using_prefix() handles a prefix that contains multiple
-- clauses for the partition key b (ie, b >= 1 and b = 2) (This also tests
-- that the caller arranges clauses in that prefix in the required order)
explain (costs off) select * from rp_prefix_test3 where a >= 1 and b >= 1 and b = 2 and c = 2 and d >= 0;
- QUERY PLAN
-------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------
Seq Scan on rp_prefix_test3_p2 rp_prefix_test3
- Filter: ((a >= 1) AND (b >= 1) AND (d >= 0) AND (b = 2) AND (c = 2))
+ Filter: ((a >= 1) AND (d >= 0) AND (b = 2) AND (c = 2))
(2 rows)
drop table rp_prefix_test1;
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index fd5654df35..04f99ccf78 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -1934,21 +1934,21 @@ INSERT INTO bv1 VALUES (11, 'xxx'); -- should fail RLS check
ERROR: new row violates row-level security policy for table "b1"
INSERT INTO bv1 VALUES (12, 'xxx'); -- ok
EXPLAIN (COSTS OFF) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b);
- QUERY PLAN
------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------
Update on b1
-> Seq Scan on b1
- Filter: ((a > 0) AND (a = 4) AND ((a % 2) = 0) AND f_leak(b))
+ Filter: ((a = 4) AND ((a % 2) = 0) AND f_leak(b))
(3 rows)
UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b);
NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02
EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b);
- QUERY PLAN
------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------
Delete on b1
-> Seq Scan on b1
- Filter: ((a > 0) AND (a = 6) AND ((a % 2) = 0) AND f_leak(b))
+ Filter: ((a = 6) AND ((a % 2) = 0) AND f_leak(b))
(3 rows)
DELETE FROM bv1 WHERE a = 6 AND f_leak(b);
@@ -3085,10 +3085,10 @@ SET SESSION AUTHORIZATION regress_rls_bob;
CREATE VIEW rls_sbv WITH (security_barrier) AS
SELECT * FROM y1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) SELECT * FROM rls_sbv WHERE (a = 1);
- QUERY PLAN
-------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Seq Scan on y1
- Filter: ((a = 1) AND ((a > 2) OR ((a % 2) = 0)) AND f_leak(b))
+ Filter: ((a = 1) AND ((a % 2) = 0) AND f_leak(b))
(2 rows)
DROP VIEW rls_sbv;
diff --git a/src/test/regress/expected/self_contradictory.out b/src/test/regress/expected/self_contradictory.out
new file mode 100644
index 0000000000..3d7cdcb333
--- /dev/null
+++ b/src/test/regress/expected/self_contradictory.out
@@ -0,0 +1,2607 @@
+--
+-- SELF CONTRADICTORY
+--
+-- Create table
+--
+CREATE TABLE self_a (a INT, b INT, c INT, d INT, z BOOL);
+CREATE TABLE self_b (a VARCHAR(6));
+--INSERT INTO self_a
+--SELECT ceil(rANDom()*(1000-1)+1) ,
+--ceil(RANDOM()*(1000-1)+1) ,
+--ceil(RANDOM()*(1000-1)+1) ,
+--ceil(RANDOM()*(1000-1)+1) ,
+--(RANDOM())::int::bool
+--FROM generate_series(1 , 300);
+--Load data
+--
+INSERT INTO self_a VALUES (56 , 118 , 32 , 780 , true);
+INSERT INTO self_a VALUES (854 , 610 , 991 , 526 , false);
+INSERT INTO self_a VALUES (837 , 872 , 417 , 130 , true);
+INSERT INTO self_a VALUES (193 , 326 , 902 , 976 , false);
+INSERT INTO self_a VALUES (414 , 913 , 549 , 289 , true);
+INSERT INTO self_a VALUES (733 , 159 , 285 , 496 , false);
+INSERT INTO self_a VALUES (44 , 253 , 569 , 357 , true);
+INSERT INTO self_a VALUES (269 , 39 , 540 , 941 , true);
+INSERT INTO self_a VALUES (807 , 974 , 329 , 4 , false);
+INSERT INTO self_a VALUES (392 , 582 , 559 , 657 , true);
+INSERT INTO self_a VALUES (836 , 337 , 672 , 303 , false);
+INSERT INTO self_a VALUES (241 , 161 , 464 , 926 , true);
+INSERT INTO self_a VALUES (569 , 792 , 682 , 462 , false);
+INSERT INTO self_a VALUES (994 , 629 , 653 , 562 , true);
+INSERT INTO self_a VALUES (614 , 505 , 543 , 510 , false);
+INSERT INTO self_a VALUES (796 , 634 , 72 , 519 , false);
+INSERT INTO self_a VALUES (982 , 357 , 767 , 752 , true);
+INSERT INTO self_a VALUES (318 , 13 , 119 , 148 , false);
+INSERT INTO self_a VALUES (568 , 940 , 548 , 426 , true);
+INSERT INTO self_a VALUES (356 , 732 , 660 , 303 , true);
+INSERT INTO self_a VALUES (332 , 755 , 409 , 493 , true);
+INSERT INTO self_a VALUES (263 , 710 , 622 , 40 , false);
+INSERT INTO self_a VALUES (205 , 128 , 909 , 468 , true);
+INSERT INTO self_a VALUES (6 , 514 , 536 , 746 , false);
+INSERT INTO self_a VALUES (153 , 850 , 127 , 470 , true);
+INSERT INTO self_a VALUES (289 , 430 , 843 , 929 , true);
+INSERT INTO self_a VALUES (797 , 168 , 438 , 902 , false);
+INSERT INTO self_a VALUES (870 , 667 , 792 , 275 , false);
+INSERT INTO self_a VALUES (247 , 382 , 911 , 522 , false);
+INSERT INTO self_a VALUES (27 , 203 , 125 , 400 , false);
+INSERT INTO self_a VALUES (153 , 678 , 79 , 278 , false);
+INSERT INTO self_a VALUES (731 , 516 , 81 , 324 , false);
+INSERT INTO self_a VALUES (602 , 648 , 892 , 787 , true);
+INSERT INTO self_a VALUES (691 , 177 , 873 , 183 , false);
+INSERT INTO self_a VALUES (455 , 962 , 675 , 268 , false);
+INSERT INTO self_a VALUES (899 , 679 , 71 , 88 , false);
+INSERT INTO self_a VALUES (755 , 985 , 569 , 555 , false);
+INSERT INTO self_a VALUES (862 , 344 , 659 , 984 , true);
+INSERT INTO self_a VALUES (222 , 163 , 299 , 75 , false);
+INSERT INTO self_a VALUES (139 , 603 , 93 , 681 , false);
+INSERT INTO self_a VALUES (196 , 783 , 85 , 485 , false);
+INSERT INTO self_a VALUES (688 , 963 , 463 , 465 , true);
+INSERT INTO self_a VALUES (804 , 894 , 833 , 52 , false);
+INSERT INTO self_a VALUES (254 , 332 , 481 , 563 , true);
+INSERT INTO self_a VALUES (564 , 43 , 33 , 445 , false);
+INSERT INTO self_a VALUES (283 , 4 , 598 , 347 , true);
+INSERT INTO self_a VALUES (108 , 736 , 693 , 731 , false);
+INSERT INTO self_a VALUES (836 , 502 , 543 , 294 , false);
+INSERT INTO self_a VALUES (900 , 739 , 645 , 132 , true);
+INSERT INTO self_a VALUES (918 , 798 , 360 , 621 , false);
+INSERT INTO self_a VALUES (302 , 998 , 212 , 696 , true);
+INSERT INTO self_a VALUES (711 , 906 , 273 , 384 , false);
+INSERT INTO self_a VALUES (699 , 884 , 563 , 120 , true);
+INSERT INTO self_a VALUES (357 , 456 , 709 , 762 , true);
+INSERT INTO self_a VALUES (708 , 276 , 237 , 862 , false);
+INSERT INTO self_a VALUES (274 , 40 , 478 , 495 , true);
+INSERT INTO self_a VALUES (564 , 973 , 743 , 123 , true);
+INSERT INTO self_a VALUES (195 , 848 , 401 , 908 , false);
+INSERT INTO self_a VALUES (819 , 38 , 449 , 715 , false);
+INSERT INTO self_a VALUES (688 , 549 , 702 , 819 , false);
+INSERT INTO self_a VALUES (242 , 716 , 55 , 88 , true);
+INSERT INTO self_a VALUES (866 , 508 , 610 , 30 , false);
+INSERT INTO self_a VALUES (720 , 558 , 719 , 721 , true);
+INSERT INTO self_a VALUES (895 , 106 , 7 , 231 , false);
+INSERT INTO self_a VALUES (895 , 926 , 991 , 18 , true);
+INSERT INTO self_a VALUES (15 , 49 , 670 , 482 , true);
+INSERT INTO self_a VALUES (499 , 122 , 139 , 68 , false);
+INSERT INTO self_a VALUES (44 , 486 , 900 , 526 , false);
+INSERT INTO self_a VALUES (502 , 390 , 266 , 59 , true);
+INSERT INTO self_a VALUES (197 , 481 , 798 , 954 , true);
+INSERT INTO self_a VALUES (1000 , 512 , 245 , 310 , false);
+INSERT INTO self_a VALUES (207 , 938 , 856 , 191 , true);
+INSERT INTO self_a VALUES (666 , 883 , 47 , 664 , false);
+INSERT INTO self_a VALUES (964 , 852 , 439 , 173 , false);
+INSERT INTO self_a VALUES (294 , 262 , 404 , 259 , false);
+INSERT INTO self_a VALUES (81 , 799 , 100 , 620 , true);
+INSERT INTO self_a VALUES (633 , 272 , 145 , 644 , true);
+INSERT INTO self_a VALUES (287 , 256 , 875 , 53 , false);
+INSERT INTO self_a VALUES (962 , 216 , 696 , 315 , false);
+INSERT INTO self_a VALUES (299 , 875 , 355 , 584 , false);
+INSERT INTO self_a VALUES (645 , 249 , 655 , 667 , true);
+INSERT INTO self_a VALUES (223 , 71 , 627 , 251 , true);
+INSERT INTO self_a VALUES (677 , 849 , 978 , 114 , false);
+INSERT INTO self_a VALUES (130 , 199 , 836 , 596 , false);
+INSERT INTO self_a VALUES (710 , 775 , 416 , 215 , false);
+INSERT INTO self_a VALUES (146 , 621 , 442 , 338 , true);
+INSERT INTO self_a VALUES (924 , 349 , 600 , 116 , false);
+INSERT INTO self_a VALUES (165 , 330 , 678 , 57 , true);
+INSERT INTO self_a VALUES (593 , 187 , 258 , 974 , true);
+INSERT INTO self_a VALUES (951 , 698 , 928 , 139 , false);
+INSERT INTO self_a VALUES (322 , 965 , 961 , 464 , false);
+INSERT INTO self_a VALUES (884 , 683 , 173 , 38 , true);
+INSERT INTO self_a VALUES (701 , 443 , 368 , 748 , true);
+INSERT INTO self_a VALUES (461 , 619 , 15 , 596 , true);
+INSERT INTO self_a VALUES (526 , 364 , 607 , 397 , true);
+INSERT INTO self_a VALUES (13 , 937 , 247 , 801 , false);
+INSERT INTO self_a VALUES (538 , 82 , 145 , 879 , false);
+INSERT INTO self_a VALUES (169 , 168 , 389 , 321 , true);
+INSERT INTO self_a VALUES (405 , 769 , 699 , 908 , true);
+INSERT INTO self_a VALUES (348 , 429 , 644 , 521 , false);
+INSERT INTO self_a VALUES (838 , 877 , 729 , 7 , false);
+INSERT INTO self_a VALUES (989 , 254 , 808 , 71 , true);
+INSERT INTO self_a VALUES (699 , 140 , 379 , 557 , true);
+INSERT INTO self_a VALUES (493 , 671 , 500 , 809 , true);
+INSERT INTO self_a VALUES (177 , 403 , 995 , 70 , true);
+INSERT INTO self_a VALUES (750 , 967 , 470 , 128 , true);
+INSERT INTO self_a VALUES (955 , 473 , 813 , 315 , true);
+INSERT INTO self_a VALUES (742 , 784 , 627 , 420 , false);
+INSERT INTO self_a VALUES (403 , 717 , 646 , 329 , true);
+INSERT INTO self_a VALUES (292 , 990 , 701 , 988 , false);
+INSERT INTO self_a VALUES (79 , 78 , 834 , 834 , false);
+INSERT INTO self_a VALUES (440 , 724 , 690 , 536 , false);
+INSERT INTO self_a VALUES (707 , 875 , 351 , 404 , true);
+INSERT INTO self_a VALUES (945 , 955 , 885 , 897 , false);
+INSERT INTO self_a VALUES (108 , 476 , 668 , 405 , false);
+INSERT INTO self_a VALUES (986 , 343 , 552 , 587 , true);
+INSERT INTO self_a VALUES (629 , 22 , 667 , 504 , true);
+INSERT INTO self_a VALUES (104 , 146 , 576 , 513 , true);
+INSERT INTO self_a VALUES (356 , 991 , 397 , 396 , true);
+INSERT INTO self_a VALUES (899 , 817 , 237 , 392 , true);
+INSERT INTO self_a VALUES (240 , 393 , 590 , 310 , false);
+INSERT INTO self_a VALUES (906 , 573 , 999 , 354 , false);
+INSERT INTO self_a VALUES (907 , 66 , 936 , 59 , false);
+INSERT INTO self_a VALUES (577 , 327 , 245 , 713 , false);
+INSERT INTO self_a VALUES (254 , 271 , 113 , 58 , true);
+INSERT INTO self_a VALUES (326 , 43 , 673 , 87 , true);
+INSERT INTO self_a VALUES (867 , 954 , 699 , 647 , false);
+INSERT INTO self_a VALUES (99 , 826 , 378 , 172 , false);
+INSERT INTO self_a VALUES (158 , 375 , 950 , 802 , false);
+INSERT INTO self_a VALUES (104 , 157 , 523 , 396 , true);
+INSERT INTO self_a VALUES (902 , 125 , 613 , 44 , false);
+INSERT INTO self_a VALUES (564 , 635 , 608 , 48 , true);
+INSERT INTO self_a VALUES (753 , 539 , 692 , 111 , true);
+INSERT INTO self_a VALUES (27 , 613 , 633 , 693 , true);
+INSERT INTO self_a VALUES (859 , 855 , 512 , 28 , false);
+INSERT INTO self_a VALUES (684 , 452 , 757 , 209 , false);
+INSERT INTO self_a VALUES (198 , 591 , 500 , 940 , false);
+INSERT INTO self_a VALUES (442 , 198 , 182 , 663 , true);
+INSERT INTO self_a VALUES (797 , 965 , 909 , 98 , false);
+INSERT INTO self_a VALUES (466 , 665 , 849 , 828 , true);
+INSERT INTO self_a VALUES (615 , 689 , 767 , 773 , false);
+INSERT INTO self_a VALUES (743 , 760 , 860 , 232 , true);
+INSERT INTO self_a VALUES (86 , 444 , 683 , 217 , true);
+INSERT INTO self_a VALUES (772 , 725 , 436 , 862 , true);
+INSERT INTO self_a VALUES (34 , 820 , 627 , 320 , true);
+INSERT INTO self_a VALUES (5 , 437 , 23 , 829 , false);
+INSERT INTO self_a VALUES (697 , 222 , 752 , 984 , true);
+INSERT INTO self_a VALUES (426 , 273 , 972 , 901 , false);
+INSERT INTO self_a VALUES (782 , 610 , 78 , 267 , true);
+INSERT INTO self_a VALUES (238 , 126 , 75 , 601 , true);
+INSERT INTO self_a VALUES (385 , 443 , 912 , 100 , true);
+INSERT INTO self_a VALUES (939 , 715 , 692 , 21 , true);
+INSERT INTO self_a VALUES (108 , 291 , 894 , 132 , false);
+INSERT INTO self_a VALUES (632 , 316 , 108 , 815 , true);
+INSERT INTO self_a VALUES (849 , 847 , 786 , 601 , false);
+INSERT INTO self_a VALUES (218 , 153 , 138 , 574 , true);
+INSERT INTO self_a VALUES (947 , 673 , 758 , 151 , true);
+INSERT INTO self_a VALUES (146 , 807 , 30 , 963 , false);
+INSERT INTO self_a VALUES (748 , 222 , 931 , 664 , true);
+INSERT INTO self_a VALUES (916 , 19 , 100 , 457 , false);
+INSERT INTO self_a VALUES (921 , 287 , 386 , 201 , true);
+INSERT INTO self_a VALUES (521 , 177 , 935 , 268 , true);
+INSERT INTO self_a VALUES (58 , 222 , 195 , 749 , true);
+INSERT INTO self_a VALUES (95 , 430 , 46 , 825 , true);
+INSERT INTO self_a VALUES (290 , 121 , 520 , 759 , false);
+INSERT INTO self_a VALUES (655 , 640 , 203 , 331 , false);
+INSERT INTO self_a VALUES (30 , 950 , 632 , 526 , true);
+INSERT INTO self_a VALUES (997 , 996 , 179 , 575 , true);
+INSERT INTO self_a VALUES (523 , 881 , 463 , 602 , false);
+INSERT INTO self_a VALUES (277 , 298 , 806 , 724 , true);
+INSERT INTO self_a VALUES (601 , 467 , 772 , 305 , false);
+INSERT INTO self_a VALUES (169 , 341 , 646 , 492 , false);
+INSERT INTO self_a VALUES (739 , 718 , 98 , 513 , true);
+INSERT INTO self_a VALUES (91 , 996 , 871 , 954 , true);
+INSERT INTO self_a VALUES (248 , 367 , 704 , 899 , true);
+INSERT INTO self_a VALUES (679 , 192 , 687 , 192 , true);
+INSERT INTO self_a VALUES (840 , 429 , 830 , 93 , false);
+INSERT INTO self_a VALUES (750 , 263 , 652 , 749 , true);
+INSERT INTO self_a VALUES (835 , 478 , 266 , 402 , true);
+INSERT INTO self_a VALUES (264 , 277 , 668 , 230 , true);
+INSERT INTO self_a VALUES (55 , 662 , 473 , 776 , true);
+INSERT INTO self_a VALUES (800 , 594 , 825 , 638 , true);
+INSERT INTO self_a VALUES (549 , 29 , 951 , 607 , false);
+INSERT INTO self_a VALUES (472 , 908 , 939 , 703 , true);
+INSERT INTO self_a VALUES (536 , 969 , 814 , 303 , true);
+INSERT INTO self_a VALUES (316 , 173 , 228 , 312 , false);
+INSERT INTO self_a VALUES (584 , 885 , 329 , 479 , false);
+INSERT INTO self_a VALUES (229 , 716 , 536 , 417 , false);
+INSERT INTO self_a VALUES (177 , 769 , 282 , 339 , true);
+INSERT INTO self_a VALUES (987 , 512 , 477 , 629 , false);
+INSERT INTO self_a VALUES (744 , 252 , 336 , 551 , false);
+INSERT INTO self_a VALUES (655 , 429 , 250 , 603 , true);
+INSERT INTO self_a VALUES (965 , 733 , 863 , 363 , false);
+INSERT INTO self_a VALUES (459 , 276 , 576 , 510 , true);
+INSERT INTO self_a VALUES (626 , 986 , 829 , 345 , false);
+INSERT INTO self_a VALUES (434 , 614 , 655 , 288 , true);
+INSERT INTO self_a VALUES (570 , 17 , 675 , 416 , true);
+INSERT INTO self_a VALUES (882 , 270 , 147 , 862 , false);
+INSERT INTO self_a VALUES (593 , 669 , 145 , 386 , false);
+INSERT INTO self_a VALUES (313 , 369 , 21 , 781 , true);
+INSERT INTO self_a VALUES (869 , 116 , 133 , 611 , true);
+INSERT INTO self_a VALUES (604 , 368 , 695 , 100 , true);
+INSERT INTO self_a VALUES (943 , 847 , 120 , 483 , true);
+INSERT INTO self_a VALUES (396 , 231 , 437 , 772 , false);
+INSERT INTO self_a VALUES (583 , 177 , 580 , 506 , true);
+INSERT INTO self_a VALUES (170 , 542 , 784 , 61 , false);
+INSERT INTO self_a VALUES (978 , 493 , 12 , 236 , false);
+INSERT INTO self_a VALUES (222 , 162 , 612 , 15 , false);
+INSERT INTO self_a VALUES (585 , 138 , 641 , 274 , true);
+INSERT INTO self_a VALUES (246 , 657 , 243 , 852 , true);
+INSERT INTO self_a VALUES (62 , 869 , 96 , 25 , false);
+INSERT INTO self_a VALUES (847 , 224 , 676 , 869 , false);
+INSERT INTO self_a VALUES (211 , 709 , 672 , 470 , true);
+INSERT INTO self_a VALUES (366 , 647 , 964 , 221 , false);
+INSERT INTO self_a VALUES (980 , 847 , 37 , 587 , false);
+INSERT INTO self_a VALUES (450 , 386 , 594 , 112 , false);
+INSERT INTO self_a VALUES (729 , 463 , 931 , 920 , false);
+INSERT INTO self_a VALUES (498 , 604 , 51 , 871 , false);
+INSERT INTO self_a VALUES (464 , 727 , 491 , 891 , false);
+INSERT INTO self_a VALUES (522 , 316 , 82 , 316 , true);
+INSERT INTO self_a VALUES (32 , 27 , 138 , 337 , false);
+INSERT INTO self_a VALUES (733 , 939 , 991 , 449 , true);
+INSERT INTO self_a VALUES (221 , 235 , 992 , 855 , false);
+INSERT INTO self_a VALUES (770 , 463 , 881 , 616 , false);
+INSERT INTO self_a VALUES (558 , 738 , 71 , 74 , false);
+INSERT INTO self_a VALUES (914 , 682 , 217 , 242 , false);
+INSERT INTO self_a VALUES (524 , 444 , 281 , 455 , false);
+INSERT INTO self_a VALUES (778 , 756 , 230 , 393 , false);
+INSERT INTO self_a VALUES (114 , 63 , 606 , 47 , true);
+INSERT INTO self_a VALUES (657 , 625 , 890 , 806 , false);
+INSERT INTO self_a VALUES (988 , 903 , 337 , 739 , true);
+INSERT INTO self_a VALUES (536 , 189 , 430 , 446 , true);
+INSERT INTO self_a VALUES (357 , 44 , 70 , 638 , false);
+INSERT INTO self_a VALUES (540 , 578 , 772 , 484 , true);
+INSERT INTO self_a VALUES (567 , 975 , 658 , 611 , true);
+INSERT INTO self_a VALUES (458 , 747 , 744 , 234 , false);
+INSERT INTO self_a VALUES (695 , 291 , 666 , 535 , false);
+INSERT INTO self_a VALUES (4 , 23 , 607 , 997 , true);
+INSERT INTO self_a VALUES (345 , 401 , 923 , 576 , true);
+INSERT INTO self_a VALUES (634 , 912 , 328 , 860 , true);
+INSERT INTO self_a VALUES (104 , 807 , 925 , 747 , true);
+INSERT INTO self_a VALUES (155 , 19 , 238 , 464 , true);
+INSERT INTO self_a VALUES (569 , 117 , 844 , 808 , false);
+INSERT INTO self_a VALUES (171 , 227 , 751 , 287 , true);
+INSERT INTO self_a VALUES (196 , 762 , 443 , 759 , true);
+INSERT INTO self_a VALUES (657 , 445 , 668 , 569 , false);
+INSERT INTO self_a VALUES (372 , 215 , 899 , 368 , true);
+INSERT INTO self_a VALUES (223 , 794 , 499 , 736 , false);
+INSERT INTO self_a VALUES (433 , 94 , 321 , 359 , true);
+INSERT INTO self_a VALUES (423 , 575 , 251 , 637 , false);
+INSERT INTO self_a VALUES (508 , 734 , 783 , 781 , true);
+INSERT INTO self_a VALUES (278 , 653 , 931 , 500 , false);
+INSERT INTO self_a VALUES (726 , 993 , 9 , 298 , true);
+INSERT INTO self_a VALUES (677 , 426 , 376 , 226 , false);
+INSERT INTO self_a VALUES (967 , 639 , 659 , 731 , false);
+INSERT INTO self_a VALUES (488 , 774 , 321 , 573 , false);
+INSERT INTO self_a VALUES (65 , 909 , 5 , 331 , true);
+INSERT INTO self_a VALUES (426 , 554 , 297 , 168 , true);
+INSERT INTO self_a VALUES (460 , 846 , 500 , 416 , true);
+INSERT INTO self_a VALUES (887 , 664 , 705 , 592 , true);
+INSERT INTO self_a VALUES (98 , 83 , 24 , 146 , false);
+INSERT INTO self_a VALUES (79 , 343 , 431 , 601 , false);
+INSERT INTO self_a VALUES (127 , 942 , 871 , 783 , true);
+INSERT INTO self_a VALUES (20 , 441 , 998 , 508 , false);
+INSERT INTO self_a VALUES (267 , 62 , 303 , 239 , true);
+INSERT INTO self_a VALUES (239 , 62 , 439 , 837 , true);
+INSERT INTO self_a VALUES (89 , 686 , 252 , 569 , false);
+INSERT INTO self_a VALUES (891 , 135 , 718 , 114 , true);
+INSERT INTO self_a VALUES (606 , 246 , 74 , 214 , false);
+INSERT INTO self_a VALUES (552 , 359 , 115 , 783 , false);
+INSERT INTO self_a VALUES (712 , 930 , 700 , 976 , false);
+INSERT INTO self_a VALUES (107 , 852 , 200 , 230 , true);
+INSERT INTO self_a VALUES (554 , 263 , 759 , 69 , false);
+INSERT INTO self_a VALUES (391 , 345 , 544 , 436 , false);
+INSERT INTO self_a VALUES (12 , 425 , 493 , 606 , false);
+INSERT INTO self_a VALUES (869 , 453 , 75 , 48 , true);
+INSERT INTO self_a VALUES (455 , 58 , 920 , 372 , true);
+INSERT INTO self_a VALUES (227 , 864 , 928 , 982 , false);
+INSERT INTO self_a VALUES (6 , 513 , 218 , 928 , false);
+INSERT INTO self_a VALUES (844 , 740 , 455 , 993 , false);
+INSERT INTO self_a VALUES (850 , 418 , 336 , 521 , false);
+INSERT INTO self_a VALUES (209 , 470 , 482 , 456 , true);
+INSERT INTO self_a VALUES (177 , 631 , 900 , 590 , true);
+INSERT INTO self_a VALUES (899 , 485 , 322 , 890 , true);
+INSERT INTO self_a VALUES (733 , 33 , 64 , 48 , true);
+INSERT INTO self_a VALUES (700 , 47 , 989 , 640 , false);
+INSERT INTO self_a VALUES (252 , 571 , 608 , 239 , false);
+INSERT INTO self_a VALUES (480 , 100 , 299 , 783 , false);
+INSERT INTO self_a VALUES (40 , 341 , 956 , 192 , false);
+INSERT INTO self_a VALUES (304 , 201 , 128 , 344 , true);
+INSERT INTO self_a VALUES (150 , 276 , 114 , 435 , false);
+INSERT INTO self_a VALUES (412 , 482 , 930 , 265 , false);
+INSERT INTO self_a VALUES (219 , 597 , 726 , 347 , true);
+INSERT INTO self_a VALUES (248 , 74 , 303 , 763 , true);
+INSERT INTO self_a VALUES (402 , 604 , 706 , 483 , true);
+INSERT INTO self_a VALUES (491 , 662 , 324 , 100 , false);
+INSERT INTO self_a VALUES (680 , 890 , 651 , 440 , true);
+INSERT INTO self_a VALUES (42 , 791 , 890 , 623 , true);
+INSERT INTO self_a VALUES (144 , 166 , 928 , 774 , false);
+INSERT INTO self_a VALUES (272 , 507 , 710 , 514 , true);
+INSERT INTO self_a VALUES (NULL, NULL, NULL, NULL);
+CREATE INDEX idx_self_a ON self_a USING BTREE (a , b , c);
+ANALYZE self_a;
+--
+-- Test single attribute self-contradictory
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a < 9;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 10 AND a < 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <= 10 AND a > 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <= 10 AND a >= 10;
+ QUERY PLAN
+-------------------------------------
+ Seq Scan on self_a
+ Filter: ((a <= 10) AND (a >= 10))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 10 AND a < 10 AND a > 8 AND a <= 12;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a = 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 10 AND a = 10;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 10)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < 10 AND a = 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <= 10 AND a = 10;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 10)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = 10 AND a <> 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = 10 AND a <> 9;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 10)
+(2 rows)
+
+--
+-- Boolean type is a little tricky.
+--
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT FALSE;
+ QUERY PLAN
+----------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS TRUE) AND (z IS NOT FALSE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT UNKNOWN;
+ QUERY PLAN
+------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS TRUE) AND (z IS NOT UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS FALSE AND z IS NOT FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS FALSE AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS FALSE AND z IS NOT UNKNOWN;
+ QUERY PLAN
+-------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS FALSE) AND (z IS NOT UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS UNKNOWN AND z IS TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS UNKNOWN AND z IS NOT TRUE;
+ QUERY PLAN
+------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS UNKNOWN) AND (z IS NOT TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS UNKNOWN AND z IS NOT UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS NOT TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS NOT FALSE;
+ QUERY PLAN
+------------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z IS NOT FALSE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS TRUE;
+ QUERY PLAN
+-------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z IS TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT UNKNOWN;
+ QUERY PLAN
+--------------------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (z IS NOT UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT TRUE;
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (z IS NOT TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS FALSE;
+ QUERY PLAN
+--------------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (z IS FALSE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT UNKNOWN;
+ QUERY PLAN
+--------------------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (z IS NOT UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND NOT z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND NOT z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= FALSE AND NOT z;
+ QUERY PLAN
+--------------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (z >= false))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z;
+ QUERY PLAN
+---------------------
+ Seq Scan on self_a
+ Filter: (z AND z)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS TRUE;
+ QUERY PLAN
+-------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z IS TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT FALSE;
+ QUERY PLAN
+------------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z IS NOT FALSE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT UNKNOWN;
+ QUERY PLAN
+--------------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z IS NOT UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z >= TRUE;
+ QUERY PLAN
+-------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z >= true))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z > TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z <= TRUE;
+ QUERY PLAN
+-------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z <= true))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z < TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z < FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS NOT TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS NOT FALSE;
+ QUERY PLAN
+----------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT FALSE) AND (z >= true))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS NOT UNKNOWN;
+ QUERY PLAN
+------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT UNKNOWN) AND (z > false))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND NOT z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS NOT TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= FALSE AND z IS NOT TRUE;
+ QUERY PLAN
+----------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT TRUE) AND (z >= false))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS NOT FALSE;
+ QUERY PLAN
+----------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT FALSE) AND (z > false))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS FALSE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND NOT z;
+ QUERY PLAN
+---------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (NOT z))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z = true;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z > false;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z >= false;
+ QUERY PLAN
+--------------------------------------
+ Seq Scan on self_a
+ Filter: ((NOT z) AND (z >= false))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z >= true;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> FALSE AND NOT z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> FALSE AND z >= false;
+ QUERY PLAN
+--------------------------------
+ Seq Scan on self_a
+ Filter: (z AND (z >= false))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> FALSE AND z <= false;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Null test
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a IS NOT NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NOT NULL AND a > 10;
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a IS NOT NULL) AND (a > 10))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a = 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a > 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a <> 10;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND NOT z;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z <> TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS TRUE;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS NOT TRUE;
+ QUERY PLAN
+---------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NULL) AND (z IS NOT TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NULL) AND (z IS UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS NOT UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND NOT z;
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT NULL) AND (NOT z))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z <> TRUE;
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT NULL) AND (NOT z))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS TRUE;
+ QUERY PLAN
+---------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT NULL) AND (z IS TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS NOT TRUE;
+ QUERY PLAN
+-------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT NULL) AND (z IS NOT TRUE))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS NOT UNKNOWN;
+ QUERY PLAN
+----------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((z IS NOT NULL) AND (z IS NOT UNKNOWN))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS UNKNOWN;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Test weak restriction that have the sematic of same level will being eliminated and IS self contradictory or not
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a > 5;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 5)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a > 5 AND a < 5;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 5;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a >= 5)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 5 AND a < 5;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 4)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 4 AND a < 4;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a >= 5;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a >= 5)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a >= 5 AND a < 5;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a > 5;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 5)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a > 5 AND a < 5;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a = 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 4)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a = 4 AND a <> 4;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 3 AND a = 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 4)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 3 AND a = 4 AND a < 4;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 3 AND a = 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 4)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> 3 AND a = 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 4)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> 3 AND a = 4 AND a > 4;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Test same type coercion will be accepted
+--
+explain (costs off)
+select * from self_a WHERE a = 1 AND a <>'1'::int;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Test different type coercion will be ignored
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a = 1 AND a <>'1'::bigint;
+ QUERY PLAN
+--------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a <> '1'::bigint) AND (a = 1))
+(2 rows)
+
+--
+-- Test specified collation will be ignored
+--
+explain (costs off)
+SELECT * FROM self_b WHERE a > 'c' AND a < 'b' COLLATE "pg_c_utf8";
+ QUERY PLAN
+-----------------------------------------------------------------------------------
+ Seq Scan on self_b
+ Filter: (((a)::text > 'c'::text) AND ((a)::text < 'b'::text COLLATE pg_c_utf8))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_b WHERE a > 'c' AND a < 'b';
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Redundant clause will be eliminated
+--
+explain (costs off)
+select * from self_a WHERE a = 1 AND a = 1;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 1)
+(2 rows)
+
+explain (costs off)
+select * from self_a WHERE a > 10 AND a > 10;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 10)
+(2 rows)
+
+explain (costs off)
+select * from self_a WHERE a >= 10 AND a >= 10;
+ QUERY PLAN
+---------------------
+ Seq Scan on self_a
+ Filter: (a >= 10)
+(2 rows)
+
+explain (costs off)
+select * from self_a WHERE a < 4 AND a < 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a < 4)
+(2 rows)
+
+explain (costs off)
+select * from self_a WHERE a <= 4 AND a <= 4;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a <= 4)
+(2 rows)
+
+--
+-- Test base or-clause
+--
+--
+-- Test or-clause self-contradictory
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a < 9 OR b > 8 AND b < 6;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a < 9 OR ((b > 8 AND b < 7 or b = 10) AND b < 6);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Test or-clause can be pruned
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a < 9 AND a > 10 OR b < 8;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (b < 8)
+(2 rows)
+
+SELECT * FROM self_a WHERE a < 9 AND a > 10 OR b < 8;
+ a | b | c | d | z
+-----+---+-----+-----+---
+ 283 | 4 | 598 | 347 | t
+(1 row)
+
+--
+-- Test or-clause redundant clause will be eliminated
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a < 14 AND a < 10 OR b < 8;
+ QUERY PLAN
+---------------------------------
+ Seq Scan on self_a
+ Filter: ((a < 10) OR (b < 8))
+(2 rows)
+
+SELECT * FROM self_a WHERE a < 14 AND a < 10 OR b < 8;
+ a | b | c | d | z
+-----+-----+-----+-----+---
+ 6 | 514 | 536 | 746 | f
+ 283 | 4 | 598 | 347 | t
+ 5 | 437 | 23 | 829 | f
+ 4 | 23 | 607 | 997 | t
+ 6 | 513 | 218 | 928 | f
+(5 rows)
+
+--
+-- Test same level and-clause whether can be push down into or-clause to test self-contradictory and can be pruned
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a < 9 OR b < 11);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a < 9 AND b < 11 OR b <=16);
+ QUERY PLAN
+-------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a > 10) AND (b > 12) AND (b <= 16))
+(2 rows)
+
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a < 9 AND b < 11 OR b <=16);
+ a | b | c | d | z
+-----+----+-----+-----+---
+ 318 | 13 | 119 | 148 | f
+(1 row)
+
+--
+-- Test mixed expr clause
+--
+explain (costs off)
+SELECT * FROM self_a WHERE NOT(a = 1 AND a <>1) AND a = 1;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 1)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a IS NOT NULL OR a = 1;
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 1)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a IS NOT NULL OR a = 1 AND a > 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT TRUE OR a = 1 AND a > 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT TRUE OR z IS NULL AND z IS NOT NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a BETWEEN 10 AND 5;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE (a > 4::bigint AND a > 4 AND a < 4 OR b > 3 AND b < 2) OR (a < 20);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a < 20)
+(2 rows)
+
+SELECT * FROM self_a WHERE (a > 1 AND a <1 OR b < 10 ) AND z IS TRUE OR b < 20 or z IS null;
+ a | b | c | d | z
+-----+----+-----+-----+---
+ 318 | 13 | 119 | 148 | f
+ 283 | 4 | 598 | 347 | t
+ 916 | 19 | 100 | 457 | f
+ 570 | 17 | 675 | 416 | t
+ 155 | 19 | 238 | 464 | t
+ | | | |
+(6 rows)
+
+--
+-- Test pruned and self-contradictory mixed case
+--
+explain (costs off)
+SELECT * FROM self_a WHERE (a > 10 AND a > 12 OR b > 8) AND b < 7 AND b < 15;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on self_a
+ Filter: ((b < 7) AND (a > 12))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a > 10 AND a < 9 OR b < 11 OR a < 7);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (((a < 9 OR b < 11) OR (b < 15 AND b < 17 OR a <12)) OR b >=8 AND a < 14);
+ QUERY PLAN
+-----------------------------------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a > 10) AND (b > 12) AND ((b < 15) OR (a < 12) OR ((b > 12) AND (a < 14))))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> 3 AND a <> 4 AND a <> 8 AND (b > 12 AND c < 80 OR c = 16 AND (a = 4 OR a = 8));
+ QUERY PLAN
+--------------------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a <> 3) AND (a <> 4) AND (a <> 8) AND (b > 12) AND (c < 80))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS not null AND (b > 12 AND c < 80 OR c = 16 AND (a IS null OR a = 8));
+ QUERY PLAN
+-------------------------------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a IS NOT NULL) AND (((b > 12) AND (c < 80)) OR ((c = 16) AND (a = 8))))
+(2 rows)
+
+--
+-- Test self-contradictory join case
+--
+explain (costs off)
+SELECT * FROM self_a a LEFT JOIN self_a b ON a.a = b.a WHERE b.a > 300 AND a.a = 145;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+select * from self_a a JOIN self_a b ON a.a =b.a WHERE a.a= 145 AND b.a < 79;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--
+-- Test row comparison
+-- It's need to add define DEBUG_ROW_DECODE for debug
+--
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b) > (NULL , 3);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) < (100 , null , 3);
+ QUERY PLAN
+---------------------
+ Seq Scan on self_a
+ Filter: (a < 100)
+(2 rows)
+
+--Equal expression
+--a<100
+--or a=100 and b<null
+--or a=100 and b=null and c<3
+--After converted
+--a<100
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) < (100 , null , 3) AND a = 101;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (1 , 2 , null);
+ QUERY PLAN
+-----------------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) > ROW(1, 2, NULL::integer))
+(2 rows)
+
+--Equal expression
+--a>1
+--or a=1 AND b>2
+--or a=1 AND b=2 AND c>null
+--After converted
+--a>1
+--or a=1 AND b>2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (1 , 2 , null) AND a < 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(NULL , b) > (3 , 3);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , null , b) > (2 , 3 , 4);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 2)
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 and null>3
+--or a=2 and null=3 and b>4
+--After converted
+--a>2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , null , b) > (2 , 3 , 4) AND a = 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , null) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, NULL::integer) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 and b>3
+--or a=2 and b=3 and null>=4
+--After converted
+--a>2
+--or a=2 and b>3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , null) >= (2 , 3 , 4) AND a < 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , null) > (2 , 3 , 4);
+ QUERY PLAN
+-----------------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, NULL::integer) > ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 and b>3
+--or a=2 and b=3 and null>4
+--After converted
+--a>2
+--or a=2 and b>3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) >= ROW(a, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND b>3
+--or a=a AND b=3 AND c>=4
+--After converted
+--a is not null and b>3
+--or a IS not null and b=3 AND c>=4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , 3 , 4) AND a IS NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (a , 3 , 4);
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) > ROW(a, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND b>3
+--or a=a AND b=3 AND c>4
+--After converted
+--a is not null and b>3
+--or a IS not null and b=3 AND c>4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) >= ROW(2, b, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 and b>b
+--or a=2 AND b=b AND c >=4
+--After converted
+--a>2
+--or a=2 AND b IS not null AND c >=4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , 4) AND a = 2 AND c < 4;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , b , 4);
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) > ROW(2, b, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 and b>b
+--or a=2 AND b=b AND c >4
+--After converted
+--a>2
+--or a=2 AND b IS not null AND c >4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , 3 , c);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) >= ROW(2, 3, c))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND c>=c
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND c IS not null
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , 3 , c) AND a = 2 AND c = 3 AND c IS NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , 3 , c);
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) > ROW(2, 3, c))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND c>c
+--After converted
+--a>2
+--or a=2 AND b>3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , b , c);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 2)
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>b
+--or a=2 AND b=b AND c>c
+--After converted
+--a>2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , b , c) AND a = 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , c);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) >= ROW(2, b, c))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>b
+--or a=2 AND b=b AND c>=c
+--After converted
+--a>2
+--or a=2 AND b=b AND c>=c
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , c) AND a = 2 AND b IS NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (a , b , c);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND b>b
+--or a=a AND b=b AND c>c
+--After converted
+--false
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , b , c);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, c) >= ROW(a, b, c))
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND b>b
+--or a=a AND b=b AND c>=c
+--After converted
+--a=a AND b=b AND c>=c
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , b , c) AND a IS NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , b , c) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(2, b, c) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--2>2
+--or 2=2 AND b>3
+--or 2=2 AND b=3 AND c>=4
+--After converted
+--b>3
+--or b=3 AND c>=4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , b , c) >= (2 , 3 , 4) AND c < 4 AND b = 3;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , b , c) > (2 , 3 , 4);
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(2, b, c) > ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--2>2
+--or 2=2 AND b>3
+--or 2=2 AND b=3 AND c>=4
+--After converted
+--b>3
+--or b=3 AND c>4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , c) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, 3, c) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND 3>3
+--or a=2 AND 3=3 AND c>=4
+--After converted
+--a>2
+--or a=2 AND c>=4
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , c) >= (2 , 3 , 4) AND c < 4 AND a = 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, 4) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 4>=4
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) >= (2 , 3 , 4) AND b = 3 AND a = 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) > (2 , 3 , 4);
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, 4) > ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 4>4
+--After converted
+--a>2
+--or a=2 AND b>3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) > (2 , 3 , 4) AND a = 2 AND b = 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, 3, 4) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND 3>3
+--or a=2 AND 3=3 AND 4>=4
+--After converted
+--a>2
+--or a=2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (2 , 3 , 4) AND a = 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) > (2 , 3 , 4);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 2)
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND 3>3
+--or a=2 AND 3=3 AND 4>4
+--After converted
+--a>2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) > (2 , 3 , 4) AND a = 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , 3 , 4) >= (2 , 3 , 4);
+ QUERY PLAN
+---------------------------------------------------
+ Result
+ One-Time Filter: (ROW(2, 3, 4) >= ROW(2, 3, 4))
+ -> Seq Scan on self_a
+(3 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , 3 , 4) > (2 , 3 , 4);
+ QUERY PLAN
+--------------------------------------------------
+ Result
+ One-Time Filter: (ROW(2, 3, 4) > ROW(2, 3, 4))
+ -> Seq Scan on self_a
+(3 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) > (a , 3 , 4);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND 3>3
+--or a=a AND 3=3 AND 4>4
+--After converted
+--false
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (a , 3 , 4);
+ QUERY PLAN
+---------------------------
+ Seq Scan on self_a
+ Filter: (a IS NOT NULL)
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND 3>3
+--or a=a AND 3=3 AND 4>=4
+--After converted
+--a IS NOT NULL
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (a , 3 , 4) AND a IS NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , b) > (4 , a , b);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--After converted
+--false
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , b) >= (4 , a , b);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(4, a, b) >= ROW(4, a, b))
+(2 rows)
+
+--Equal expression
+--4>4
+--or 4=4 AND a>a
+--or 4=4 AND a=a AND b>=b
+--After converted
+--a IS NOT NULL and b IS NOT NULL
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , b) >= (4 , a , b) AND a IS NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 4 , b) >= (a , 4 , b);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, 4, b) >= ROW(a, 4, b))
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND 4>4
+--or a=a AND 4=4 AND b>=b
+--After converted
+--a IS NOT NULL and b IS NOT NULL
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , 4) >= (4 , a , 4);
+ QUERY PLAN
+---------------------------
+ Seq Scan on self_a
+ Filter: (a IS NOT NULL)
+(2 rows)
+
+--Equal expression
+--4>4
+--or 4=4 AND a>a
+--or 4=4 AND a=a AND 4>=4
+--After converted
+--a IS NOT NULL
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , a , a) >= (a , a , a);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, a, a) >= ROW(a, a, a))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , a , a) > (a , a , a);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Test const are not equal
+--negative
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(1 , b , c) >= (2 , 3 , 4);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Equal expression
+--1>2
+--or 1=2 AND b>3
+--or 1=2 AND b=3 AND c>=4
+--After converted
+--false
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(1 , b , c) > (2 , 3 , 4);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Equal expression
+--1>2
+--or 1=2 AND b>3
+--or 1=2 AND b=3 AND c>4
+--After converted
+--false
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , c) >= (2 , 3 , 4);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a > 2)
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND 2>3
+--or a=2 AND 2=3 AND c>=4
+--After converted
+--a>2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , c) >= (2 , 3 , 4) AND a = 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, 3) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 3>=4
+--After converted
+--a>2
+--or a=2 AND b>3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3) >= (2 , 3 , 4) AND a < 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3 , c) >= (2 , 3 , 4 , 5);
+ QUERY PLAN
+------------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, 3, c) >= ROW(2, 3, 4, 5))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 3>=4
+--or a=2 AND b=3 AND 3=4 and c>=5
+--After converted
+--a>2
+--or a=2 AND b>3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3) >= (a , b , 4);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND b>b
+--or a=a AND b=b AND 3>=4
+--After converted
+--false
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 5 , b , 3) >= (a , 5 , b , 4);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Equal expression
+--a>a
+--or a=a AND 5>5
+--or a=a AND 5=5 and b>b
+--or a=a AND 5=5 and b=b and 3>=4
+--After converted
+--false
+--positive
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(5 , b , c) > (2 , 3 , 4);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+(1 row)
+
+--Equal expression
+--5>2
+--or 5=2 AND b>3
+--or 5=2 AND b=3 AND c>4
+--After converted
+--true
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 5 , c) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, 5, c) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND 5>3
+--or a=2 AND 5=3 AND c>=4
+--After converted
+--a>2
+--or a=2
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 5 , c) >= (2 , 3 , 4) AND a = 1;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 5) >= (2 , 3 , 4);
+ QUERY PLAN
+------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, 5) >= ROW(2, 3, 4))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 5>=4
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 5) >= (2 , 3 , 4) AND a = 2 AND b < 3;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 5 , c) >= (2 , 3 , 4 , 6);
+ QUERY PLAN
+------------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, b, 5, c) >= ROW(2, 3, 4, 6))
+(2 rows)
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 5>=4
+--or a=2 AND b=3 AND 5=4 and c>=6
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , b , 4 , 3 , c) >= (108 , 2 , b , 2 , 4 , 5);
+ QUERY PLAN
+--------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: (ROW(a, 2, b, 4, 3, c) >= ROW(108, 2, b, 2, 4, 5))
+(2 rows)
+
+--Equal expression
+--a>108
+--or a=108 and 2>2
+--or a=108 and 2=2 and b>b
+--or a=108 and 2=2 and b=b and 4>2
+--or a=108 and 2=2 and b=b and 4=2 and 3>4
+--or a=108 and 2=2 and b=b and 4=2 and 3=4 and c>=5
+--After converted
+--a>108
+--or a=108 and b IS NOT NULL
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , b , 4 , 3 , c) >= (108 , 2 , b , 2 , 4 , 5) AND a = 108 AND b is NULL;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Only keep expr that is simple after converted.
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b, c) > (23, 34, 56) AND b = 34 AND c = 56;
+ QUERY PLAN
+------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a > 23) AND (b = 34) AND (c = 56))
+(2 rows)
+
+--Test row comparison under the or-expr is const true and the or-expr will be const true.
+explain (costs off)
+SELECT * FROM self_a WHERE (ROW(2,a,b) > (1,3,5) OR c < 345 AND a = 89) AND a < 150 OR c = 178;
+ QUERY PLAN
+------------------------------------
+ Seq Scan on self_a
+ Filter: ((a < 150) OR (c = 178))
+(2 rows)
+
+--
+-- Test scalar array operation expr
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND a = 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND a < 20;
+ QUERY PLAN
+---------------------------------------------
+ Bitmap Heap Scan on self_a
+ Recheck Cond: (a < 20)
+ Filter: (a <> ALL ('{2,3,4}'::integer[]))
+ -> Bitmap Index Scan on idx_self_a
+ Index Cond: (a < 20)
+(5 rows)
+
+SELECT COUNT(*) FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND a < 20;
+ count
+-------
+ 6
+(1 row)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND (a = 2 or a = 4);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 2]) AND a = 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ANY (array[2 , 2]) AND a = 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ALL (array[20 , 30 , 40]) AND a = 27;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ALL (array[20 , 30 , 40]) AND a > 25;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ALL (array[20 , 20]) AND a >= 25;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[20 , 30 , 40]) AND a >= 25;
+ QUERY PLAN
+----------------------------------------------
+ Bitmap Heap Scan on self_a
+ Recheck Cond: ((a >= 25) AND (a < 40))
+ -> Bitmap Index Scan on idx_self_a
+ Index Cond: ((a >= 25) AND (a < 40))
+(4 rows)
+
+SELECT COUNT(*) FROM self_a WHERE a < ANY (array[20 , 30 , 40]) AND a >= 25;
+ count
+-------
+ 5
+(1 row)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[20 , 30 , 40]) AND a > 40;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[20 , 20]) AND a > 20;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[2 , 3]);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[54, NULL]);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[2 , 2]) AND a <> 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[27 , 27]) AND (a <> 27 OR a =27);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 27)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ANY (array[2]) AND a <> 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ANY (array[2 , 2]) AND a <> 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ANY (array[2 , 3 , 4]) AND a < 2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (2 , 3 , 4) AND a IN (5);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[1 , 2]) AND a > ANY (array[2 , 3]);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ALL (array[2 , 3]) AND a <=3;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ANY (array[2 , 3]) AND a <=3;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on self_a
+ Filter: ((a <= 3) AND (a > 2))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ANY (array[2 , 3]) AND a <=2;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (58 , 32 , 4) AND a IN (1 , 32 , 6);
+ QUERY PLAN
+--------------------
+ Seq Scan on self_a
+ Filter: (a = 32)
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (58 , 32 , 4) AND a IN (1 , 3 , 6);
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (1 , 2 , 3) AND a IN (1 , 3 , 6) AND a > 1;
+ QUERY PLAN
+---------------------------------
+ Seq Scan on self_a
+ Filter: ((a > 1) AND (a = 3))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ANY (array[2 , 3]) AND a <=2 OR a = ANY (array[2 , 3]) AND a > 4;
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+-- Test push scalar array expr down.
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (89, 12, 219) AND (b > 50 OR c < 300 AND a IN (89, 121, 319));
+ QUERY PLAN
+---------------------------------------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a = ANY ('{89,12,219}'::integer[])) AND ((b > 50) OR ((c < 300) AND (a = 89))))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (89, 12, 219) AND (b > 50 OR (c < 345 AND c < 78 OR a IN (89, 121, 319)));
+ QUERY PLAN
+-----------------------------------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a = ANY ('{89,12,219}'::integer[])) AND ((b > 50) OR (c < 78) OR (a = 89)))
+(2 rows)
+
+--
+-- Test push scalar array expr that operator is <> down.
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219) AND (b > 50 OR c < 345 AND a = 89);
+ QUERY PLAN
+----------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((b > 50) AND (a <> ALL ('{89,12,219}'::integer[])))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219) AND (b > 50 OR (c < 345 AND c < 78 OR a = 89));
+ QUERY PLAN
+------------------------------------------------------------------------------
+ Seq Scan on self_a
+ Filter: ((a <> ALL ('{89,12,219}'::integer[])) AND ((b > 50) OR (c < 78)))
+(2 rows)
+
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219) AND (b > 50 and a = 12 OR (c < 345 AND c < 78 and a = 219 OR a = 89));
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
+--Test using binary search.
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219, 78, 69) AND (b > 50 and a = 12 OR (c < 345 AND c < 78 and a = 219 OR a = 89));
+ QUERY PLAN
+--------------------------
+ Result
+ One-Time Filter: false
+(2 rows)
+
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index a4c7be487e..48114881a3 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1219,19 +1219,19 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a < ANY (ARRAY[1, 51]) AND b > ''1''');
estimated | actual
-----------+--------
- 2472 | 2400
+ 2448 | 2400
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a >= ANY (ARRAY[1, 51]) AND b <= ANY (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 1441 | 1250
+ 1287 | 1250
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a <= ANY (ARRAY[1, 2, 51, 52]) AND b >= ANY (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 3909 | 2550
+ 2597 | 2550
(1 row)
-- ALL (should not benefit from functional dependencies)
@@ -1244,13 +1244,13 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a IN (1, 51) AND b = ALL (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 1 | 0
+ 0 | 0
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a IN (1, 2, 51, 52) AND b = ALL (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 1 | 0
+ 0 | 0
(1 row)
-- create statistics
@@ -1385,38 +1385,38 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a < ANY (ARRAY[1, 51]) AND b > ''1''');
estimated | actual
-----------+--------
- 2472 | 2400
+ 2448 | 2400
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a >= ANY (ARRAY[1, 51]) AND b <= ANY (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 1441 | 1250
+ 1287 | 1250
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a <= ANY (ARRAY[1, 2, 51, 52]) AND b >= ANY (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 3909 | 2550
+ 2597 | 2550
(1 row)
-- ALL (should not benefit from functional dependencies)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a IN (1, 51) AND b = ALL (ARRAY[''1''])');
estimated | actual
-----------+--------
- 2 | 100
+ 100 | 100
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a IN (1, 51) AND b = ALL (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 1 | 0
+ 0 | 0
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE a IN (1, 2, 51, 52) AND b = ALL (ARRAY[''1'', ''2''])');
estimated | actual
-----------+--------
- 1 | 0
+ 0 | 0
(1 row)
-- changing the type of column c causes all its stats to be dropped, reverting
@@ -2032,37 +2032,37 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = ANY (ARRAY
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a <= ANY (ARRAY[1, 2, 3]) AND b IN (''1'', ''2'', ''3'')');
estimated | actual
-----------+--------
- 26 | 150
+ 12 | 150
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a <= ANY (ARRAY[1, NULL, 2, 3]) AND b IN (''1'', ''2'', NULL, ''3'')');
estimated | actual
-----------+--------
- 26 | 150
+ 12 | 150
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < ALL (ARRAY[4, 5]) AND c > ANY (ARRAY[1, 2, 3])');
estimated | actual
-----------+--------
- 10 | 100
+ 184 | 100
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < ALL (ARRAY[4, 5]) AND c > ANY (ARRAY[1, 2, 3, NULL])');
estimated | actual
-----------+--------
- 10 | 100
+ 184 | 100
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < ALL (ARRAY[4, 5]) AND b IN (''1'', ''2'', ''3'') AND c > ANY (ARRAY[1, 2, 3])');
estimated | actual
-----------+--------
- 1 | 100
+ 11 | 100
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < ALL (ARRAY[4, 5]) AND b IN (''1'', ''2'', NULL, ''3'') AND c > ANY (ARRAY[1, 2, NULL, 3])');
estimated | actual
-----------+--------
- 1 | 100
+ 11 | 100
(1 row)
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = ANY (ARRAY[4,5]) AND 4 = ANY(ia)');
diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out
index 8786058ed0..8f6e59f054 100644
--- a/src/test/regress/expected/updatable_views.out
+++ b/src/test/regress/expected/updatable_views.out
@@ -535,7 +535,7 @@ EXPLAIN (costs off) UPDATE rw_view1 SET a=6 WHERE a=5;
--------------------------------------------------
Update on base_tbl
-> Index Scan using base_tbl_pkey on base_tbl
- Index Cond: ((a > 0) AND (a = 5))
+ Index Cond: (a = 5)
(3 rows)
EXPLAIN (costs off) DELETE FROM rw_view1 WHERE a=5;
@@ -543,7 +543,7 @@ EXPLAIN (costs off) DELETE FROM rw_view1 WHERE a=5;
--------------------------------------------------
Delete on base_tbl
-> Index Scan using base_tbl_pkey on base_tbl
- Index Cond: ((a > 0) AND (a = 5))
+ Index Cond: (a = 5)
(3 rows)
EXPLAIN (costs off)
@@ -553,7 +553,7 @@ MERGE INTO rw_view1 t USING (VALUES (5, 'X')) AS v(a,b) ON t.a = v.a
--------------------------------------------------
Merge on base_tbl
-> Index Scan using base_tbl_pkey on base_tbl
- Index Cond: ((a > 0) AND (a = 5))
+ Index Cond: (a = 5)
(3 rows)
EXPLAIN (costs off)
@@ -728,19 +728,19 @@ SELECT * FROM rw_view2 ORDER BY aaa;
(3 rows)
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
- QUERY PLAN
---------------------------------------------------------
+ QUERY PLAN
+--------------------------------------------------
Update on base_tbl
-> Index Scan using base_tbl_pkey on base_tbl
- Index Cond: ((a < 10) AND (a > 0) AND (a = 4))
+ Index Cond: (a = 4)
(3 rows)
EXPLAIN (costs off) DELETE FROM rw_view2 WHERE aaa=4;
- QUERY PLAN
---------------------------------------------------------
+ QUERY PLAN
+--------------------------------------------------
Delete on base_tbl
-> Index Scan using base_tbl_pkey on base_tbl
- Index Cond: ((a < 10) AND (a > 0) AND (a = 4))
+ Index Cond: (a = 4)
(3 rows)
DROP TABLE base_tbl CASCADE;
@@ -924,14 +924,14 @@ MERGE INTO rw_view2 t USING (VALUES (3, 'Row 3')) AS v(a,b) ON t.a = v.a
ERROR: cannot execute MERGE on relation "rw_view1"
DETAIL: MERGE is not supported for relations with rules.
EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
- QUERY PLAN
-----------------------------------------------------------------
+ QUERY PLAN
+------------------------------------------------------------
Update on base_tbl
-> Nested Loop
-> Index Scan using base_tbl_pkey on base_tbl
Index Cond: (a = 2)
-> Subquery Scan on rw_view1
- Filter: ((rw_view1.a < 10) AND (rw_view1.a = 2))
+ Filter: (rw_view1.a = 2)
-> Bitmap Heap Scan on base_tbl base_tbl_1
Recheck Cond: (a > 0)
-> Bitmap Index Scan on base_tbl_pkey
@@ -939,14 +939,14 @@ EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
(10 rows)
EXPLAIN (costs off) DELETE FROM rw_view2 WHERE a=2;
- QUERY PLAN
-----------------------------------------------------------------
+ QUERY PLAN
+------------------------------------------------------------
Delete on base_tbl
-> Nested Loop
-> Index Scan using base_tbl_pkey on base_tbl
Index Cond: (a = 2)
-> Subquery Scan on rw_view1
- Filter: ((rw_view1.a < 10) AND (rw_view1.a = 2))
+ Filter: (rw_view1.a = 2)
-> Bitmap Heap Scan on base_tbl base_tbl_1
Recheck Cond: (a > 0)
-> Bitmap Index Scan on base_tbl_pkey
@@ -1202,11 +1202,11 @@ SELECT * FROM base_tbl ORDER BY a;
(6 rows)
EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
- QUERY PLAN
-----------------------------------------------------------
+ QUERY PLAN
+------------------------------------------------------
Update on rw_view1 rw_view1_1
-> Subquery Scan on rw_view1
- Filter: ((rw_view1.a < 10) AND (rw_view1.a = 2))
+ Filter: (rw_view1.a = 2)
-> Bitmap Heap Scan on base_tbl
Recheck Cond: (a > 0)
-> Bitmap Index Scan on base_tbl_pkey
@@ -1214,11 +1214,11 @@ EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
(7 rows)
EXPLAIN (costs off) DELETE FROM rw_view2 WHERE a=2;
- QUERY PLAN
-----------------------------------------------------------
+ QUERY PLAN
+------------------------------------------------------
Delete on rw_view1 rw_view1_1
-> Subquery Scan on rw_view1
- Filter: ((rw_view1.a < 10) AND (rw_view1.a = 2))
+ Filter: (rw_view1.a = 2)
-> Bitmap Heap Scan on base_tbl
Recheck Cond: (a > 0)
-> Bitmap Index Scan on base_tbl_pkey
@@ -3265,7 +3265,7 @@ UPDATE v1 SET a=a+1 WHERE snoop(a) AND leakproof(a) AND a = 8;
-> Append
-> Index Scan using t1_a_idx on public.t1 t1_1
Output: t1_1.a, t1_1.tableoid, t1_1.ctid
- Index Cond: ((t1_1.a > 5) AND (t1_1.a = 8))
+ Index Cond: (t1_1.a = 8)
Filter: (EXISTS(SubPlan 1) AND snoop(t1_1.a) AND leakproof(t1_1.a))
SubPlan 1
-> Append
@@ -3275,15 +3275,15 @@ UPDATE v1 SET a=a+1 WHERE snoop(a) AND leakproof(a) AND a = 8;
Filter: (t12_2.a = t1_1.a)
-> Index Scan using t11_a_idx on public.t11 t1_2
Output: t1_2.a, t1_2.tableoid, t1_2.ctid
- Index Cond: ((t1_2.a > 5) AND (t1_2.a = 8))
+ Index Cond: (t1_2.a = 8)
Filter: (EXISTS(SubPlan 1) AND snoop(t1_2.a) AND leakproof(t1_2.a))
-> Index Scan using t12_a_idx on public.t12 t1_3
Output: t1_3.a, t1_3.tableoid, t1_3.ctid
- Index Cond: ((t1_3.a > 5) AND (t1_3.a = 8))
+ Index Cond: (t1_3.a = 8)
Filter: (EXISTS(SubPlan 1) AND snoop(t1_3.a) AND leakproof(t1_3.a))
-> Index Scan using t111_a_idx on public.t111 t1_4
Output: t1_4.a, t1_4.tableoid, t1_4.ctid
- Index Cond: ((t1_4.a > 5) AND (t1_4.a = 8))
+ Index Cond: (t1_4.a = 8)
Filter: (EXISTS(SubPlan 1) AND snoop(t1_4.a) AND leakproof(t1_4.a))
(30 rows)
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 81e4222d26..a48841601a 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -136,3 +136,6 @@ test: fast_default
# run tablespace test at the end because it drops the tablespace created during
# setup that other tests may use.
test: tablespace
+
+# run self-contradictory checking test
+test: self_contradictory
\ No newline at end of file
diff --git a/src/test/regress/sql/self_contradictory.sql b/src/test/regress/sql/self_contradictory.sql
new file mode 100644
index 0000000000..921cef335e
--- /dev/null
+++ b/src/test/regress/sql/self_contradictory.sql
@@ -0,0 +1,1481 @@
+--
+-- SELF CONTRADICTORY
+--
+
+-- Create table
+--
+CREATE TABLE self_a (a INT, b INT, c INT, d INT, z BOOL);
+CREATE TABLE self_b (a VARCHAR(6));
+
+--INSERT INTO self_a
+--SELECT ceil(rANDom()*(1000-1)+1) ,
+--ceil(RANDOM()*(1000-1)+1) ,
+--ceil(RANDOM()*(1000-1)+1) ,
+--ceil(RANDOM()*(1000-1)+1) ,
+--(RANDOM())::int::bool
+--FROM generate_series(1 , 300);
+
+--Load data
+--
+
+INSERT INTO self_a VALUES (56 , 118 , 32 , 780 , true);
+INSERT INTO self_a VALUES (854 , 610 , 991 , 526 , false);
+INSERT INTO self_a VALUES (837 , 872 , 417 , 130 , true);
+INSERT INTO self_a VALUES (193 , 326 , 902 , 976 , false);
+INSERT INTO self_a VALUES (414 , 913 , 549 , 289 , true);
+INSERT INTO self_a VALUES (733 , 159 , 285 , 496 , false);
+INSERT INTO self_a VALUES (44 , 253 , 569 , 357 , true);
+INSERT INTO self_a VALUES (269 , 39 , 540 , 941 , true);
+INSERT INTO self_a VALUES (807 , 974 , 329 , 4 , false);
+INSERT INTO self_a VALUES (392 , 582 , 559 , 657 , true);
+INSERT INTO self_a VALUES (836 , 337 , 672 , 303 , false);
+INSERT INTO self_a VALUES (241 , 161 , 464 , 926 , true);
+INSERT INTO self_a VALUES (569 , 792 , 682 , 462 , false);
+INSERT INTO self_a VALUES (994 , 629 , 653 , 562 , true);
+INSERT INTO self_a VALUES (614 , 505 , 543 , 510 , false);
+INSERT INTO self_a VALUES (796 , 634 , 72 , 519 , false);
+INSERT INTO self_a VALUES (982 , 357 , 767 , 752 , true);
+INSERT INTO self_a VALUES (318 , 13 , 119 , 148 , false);
+INSERT INTO self_a VALUES (568 , 940 , 548 , 426 , true);
+INSERT INTO self_a VALUES (356 , 732 , 660 , 303 , true);
+INSERT INTO self_a VALUES (332 , 755 , 409 , 493 , true);
+INSERT INTO self_a VALUES (263 , 710 , 622 , 40 , false);
+INSERT INTO self_a VALUES (205 , 128 , 909 , 468 , true);
+INSERT INTO self_a VALUES (6 , 514 , 536 , 746 , false);
+INSERT INTO self_a VALUES (153 , 850 , 127 , 470 , true);
+INSERT INTO self_a VALUES (289 , 430 , 843 , 929 , true);
+INSERT INTO self_a VALUES (797 , 168 , 438 , 902 , false);
+INSERT INTO self_a VALUES (870 , 667 , 792 , 275 , false);
+INSERT INTO self_a VALUES (247 , 382 , 911 , 522 , false);
+INSERT INTO self_a VALUES (27 , 203 , 125 , 400 , false);
+INSERT INTO self_a VALUES (153 , 678 , 79 , 278 , false);
+INSERT INTO self_a VALUES (731 , 516 , 81 , 324 , false);
+INSERT INTO self_a VALUES (602 , 648 , 892 , 787 , true);
+INSERT INTO self_a VALUES (691 , 177 , 873 , 183 , false);
+INSERT INTO self_a VALUES (455 , 962 , 675 , 268 , false);
+INSERT INTO self_a VALUES (899 , 679 , 71 , 88 , false);
+INSERT INTO self_a VALUES (755 , 985 , 569 , 555 , false);
+INSERT INTO self_a VALUES (862 , 344 , 659 , 984 , true);
+INSERT INTO self_a VALUES (222 , 163 , 299 , 75 , false);
+INSERT INTO self_a VALUES (139 , 603 , 93 , 681 , false);
+INSERT INTO self_a VALUES (196 , 783 , 85 , 485 , false);
+INSERT INTO self_a VALUES (688 , 963 , 463 , 465 , true);
+INSERT INTO self_a VALUES (804 , 894 , 833 , 52 , false);
+INSERT INTO self_a VALUES (254 , 332 , 481 , 563 , true);
+INSERT INTO self_a VALUES (564 , 43 , 33 , 445 , false);
+INSERT INTO self_a VALUES (283 , 4 , 598 , 347 , true);
+INSERT INTO self_a VALUES (108 , 736 , 693 , 731 , false);
+INSERT INTO self_a VALUES (836 , 502 , 543 , 294 , false);
+INSERT INTO self_a VALUES (900 , 739 , 645 , 132 , true);
+INSERT INTO self_a VALUES (918 , 798 , 360 , 621 , false);
+INSERT INTO self_a VALUES (302 , 998 , 212 , 696 , true);
+INSERT INTO self_a VALUES (711 , 906 , 273 , 384 , false);
+INSERT INTO self_a VALUES (699 , 884 , 563 , 120 , true);
+INSERT INTO self_a VALUES (357 , 456 , 709 , 762 , true);
+INSERT INTO self_a VALUES (708 , 276 , 237 , 862 , false);
+INSERT INTO self_a VALUES (274 , 40 , 478 , 495 , true);
+INSERT INTO self_a VALUES (564 , 973 , 743 , 123 , true);
+INSERT INTO self_a VALUES (195 , 848 , 401 , 908 , false);
+INSERT INTO self_a VALUES (819 , 38 , 449 , 715 , false);
+INSERT INTO self_a VALUES (688 , 549 , 702 , 819 , false);
+INSERT INTO self_a VALUES (242 , 716 , 55 , 88 , true);
+INSERT INTO self_a VALUES (866 , 508 , 610 , 30 , false);
+INSERT INTO self_a VALUES (720 , 558 , 719 , 721 , true);
+INSERT INTO self_a VALUES (895 , 106 , 7 , 231 , false);
+INSERT INTO self_a VALUES (895 , 926 , 991 , 18 , true);
+INSERT INTO self_a VALUES (15 , 49 , 670 , 482 , true);
+INSERT INTO self_a VALUES (499 , 122 , 139 , 68 , false);
+INSERT INTO self_a VALUES (44 , 486 , 900 , 526 , false);
+INSERT INTO self_a VALUES (502 , 390 , 266 , 59 , true);
+INSERT INTO self_a VALUES (197 , 481 , 798 , 954 , true);
+INSERT INTO self_a VALUES (1000 , 512 , 245 , 310 , false);
+INSERT INTO self_a VALUES (207 , 938 , 856 , 191 , true);
+INSERT INTO self_a VALUES (666 , 883 , 47 , 664 , false);
+INSERT INTO self_a VALUES (964 , 852 , 439 , 173 , false);
+INSERT INTO self_a VALUES (294 , 262 , 404 , 259 , false);
+INSERT INTO self_a VALUES (81 , 799 , 100 , 620 , true);
+INSERT INTO self_a VALUES (633 , 272 , 145 , 644 , true);
+INSERT INTO self_a VALUES (287 , 256 , 875 , 53 , false);
+INSERT INTO self_a VALUES (962 , 216 , 696 , 315 , false);
+INSERT INTO self_a VALUES (299 , 875 , 355 , 584 , false);
+INSERT INTO self_a VALUES (645 , 249 , 655 , 667 , true);
+INSERT INTO self_a VALUES (223 , 71 , 627 , 251 , true);
+INSERT INTO self_a VALUES (677 , 849 , 978 , 114 , false);
+INSERT INTO self_a VALUES (130 , 199 , 836 , 596 , false);
+INSERT INTO self_a VALUES (710 , 775 , 416 , 215 , false);
+INSERT INTO self_a VALUES (146 , 621 , 442 , 338 , true);
+INSERT INTO self_a VALUES (924 , 349 , 600 , 116 , false);
+INSERT INTO self_a VALUES (165 , 330 , 678 , 57 , true);
+INSERT INTO self_a VALUES (593 , 187 , 258 , 974 , true);
+INSERT INTO self_a VALUES (951 , 698 , 928 , 139 , false);
+INSERT INTO self_a VALUES (322 , 965 , 961 , 464 , false);
+INSERT INTO self_a VALUES (884 , 683 , 173 , 38 , true);
+INSERT INTO self_a VALUES (701 , 443 , 368 , 748 , true);
+INSERT INTO self_a VALUES (461 , 619 , 15 , 596 , true);
+INSERT INTO self_a VALUES (526 , 364 , 607 , 397 , true);
+INSERT INTO self_a VALUES (13 , 937 , 247 , 801 , false);
+INSERT INTO self_a VALUES (538 , 82 , 145 , 879 , false);
+INSERT INTO self_a VALUES (169 , 168 , 389 , 321 , true);
+INSERT INTO self_a VALUES (405 , 769 , 699 , 908 , true);
+INSERT INTO self_a VALUES (348 , 429 , 644 , 521 , false);
+INSERT INTO self_a VALUES (838 , 877 , 729 , 7 , false);
+INSERT INTO self_a VALUES (989 , 254 , 808 , 71 , true);
+INSERT INTO self_a VALUES (699 , 140 , 379 , 557 , true);
+INSERT INTO self_a VALUES (493 , 671 , 500 , 809 , true);
+INSERT INTO self_a VALUES (177 , 403 , 995 , 70 , true);
+INSERT INTO self_a VALUES (750 , 967 , 470 , 128 , true);
+INSERT INTO self_a VALUES (955 , 473 , 813 , 315 , true);
+INSERT INTO self_a VALUES (742 , 784 , 627 , 420 , false);
+INSERT INTO self_a VALUES (403 , 717 , 646 , 329 , true);
+INSERT INTO self_a VALUES (292 , 990 , 701 , 988 , false);
+INSERT INTO self_a VALUES (79 , 78 , 834 , 834 , false);
+INSERT INTO self_a VALUES (440 , 724 , 690 , 536 , false);
+INSERT INTO self_a VALUES (707 , 875 , 351 , 404 , true);
+INSERT INTO self_a VALUES (945 , 955 , 885 , 897 , false);
+INSERT INTO self_a VALUES (108 , 476 , 668 , 405 , false);
+INSERT INTO self_a VALUES (986 , 343 , 552 , 587 , true);
+INSERT INTO self_a VALUES (629 , 22 , 667 , 504 , true);
+INSERT INTO self_a VALUES (104 , 146 , 576 , 513 , true);
+INSERT INTO self_a VALUES (356 , 991 , 397 , 396 , true);
+INSERT INTO self_a VALUES (899 , 817 , 237 , 392 , true);
+INSERT INTO self_a VALUES (240 , 393 , 590 , 310 , false);
+INSERT INTO self_a VALUES (906 , 573 , 999 , 354 , false);
+INSERT INTO self_a VALUES (907 , 66 , 936 , 59 , false);
+INSERT INTO self_a VALUES (577 , 327 , 245 , 713 , false);
+INSERT INTO self_a VALUES (254 , 271 , 113 , 58 , true);
+INSERT INTO self_a VALUES (326 , 43 , 673 , 87 , true);
+INSERT INTO self_a VALUES (867 , 954 , 699 , 647 , false);
+INSERT INTO self_a VALUES (99 , 826 , 378 , 172 , false);
+INSERT INTO self_a VALUES (158 , 375 , 950 , 802 , false);
+INSERT INTO self_a VALUES (104 , 157 , 523 , 396 , true);
+INSERT INTO self_a VALUES (902 , 125 , 613 , 44 , false);
+INSERT INTO self_a VALUES (564 , 635 , 608 , 48 , true);
+INSERT INTO self_a VALUES (753 , 539 , 692 , 111 , true);
+INSERT INTO self_a VALUES (27 , 613 , 633 , 693 , true);
+INSERT INTO self_a VALUES (859 , 855 , 512 , 28 , false);
+INSERT INTO self_a VALUES (684 , 452 , 757 , 209 , false);
+INSERT INTO self_a VALUES (198 , 591 , 500 , 940 , false);
+INSERT INTO self_a VALUES (442 , 198 , 182 , 663 , true);
+INSERT INTO self_a VALUES (797 , 965 , 909 , 98 , false);
+INSERT INTO self_a VALUES (466 , 665 , 849 , 828 , true);
+INSERT INTO self_a VALUES (615 , 689 , 767 , 773 , false);
+INSERT INTO self_a VALUES (743 , 760 , 860 , 232 , true);
+INSERT INTO self_a VALUES (86 , 444 , 683 , 217 , true);
+INSERT INTO self_a VALUES (772 , 725 , 436 , 862 , true);
+INSERT INTO self_a VALUES (34 , 820 , 627 , 320 , true);
+INSERT INTO self_a VALUES (5 , 437 , 23 , 829 , false);
+INSERT INTO self_a VALUES (697 , 222 , 752 , 984 , true);
+INSERT INTO self_a VALUES (426 , 273 , 972 , 901 , false);
+INSERT INTO self_a VALUES (782 , 610 , 78 , 267 , true);
+INSERT INTO self_a VALUES (238 , 126 , 75 , 601 , true);
+INSERT INTO self_a VALUES (385 , 443 , 912 , 100 , true);
+INSERT INTO self_a VALUES (939 , 715 , 692 , 21 , true);
+INSERT INTO self_a VALUES (108 , 291 , 894 , 132 , false);
+INSERT INTO self_a VALUES (632 , 316 , 108 , 815 , true);
+INSERT INTO self_a VALUES (849 , 847 , 786 , 601 , false);
+INSERT INTO self_a VALUES (218 , 153 , 138 , 574 , true);
+INSERT INTO self_a VALUES (947 , 673 , 758 , 151 , true);
+INSERT INTO self_a VALUES (146 , 807 , 30 , 963 , false);
+INSERT INTO self_a VALUES (748 , 222 , 931 , 664 , true);
+INSERT INTO self_a VALUES (916 , 19 , 100 , 457 , false);
+INSERT INTO self_a VALUES (921 , 287 , 386 , 201 , true);
+INSERT INTO self_a VALUES (521 , 177 , 935 , 268 , true);
+INSERT INTO self_a VALUES (58 , 222 , 195 , 749 , true);
+INSERT INTO self_a VALUES (95 , 430 , 46 , 825 , true);
+INSERT INTO self_a VALUES (290 , 121 , 520 , 759 , false);
+INSERT INTO self_a VALUES (655 , 640 , 203 , 331 , false);
+INSERT INTO self_a VALUES (30 , 950 , 632 , 526 , true);
+INSERT INTO self_a VALUES (997 , 996 , 179 , 575 , true);
+INSERT INTO self_a VALUES (523 , 881 , 463 , 602 , false);
+INSERT INTO self_a VALUES (277 , 298 , 806 , 724 , true);
+INSERT INTO self_a VALUES (601 , 467 , 772 , 305 , false);
+INSERT INTO self_a VALUES (169 , 341 , 646 , 492 , false);
+INSERT INTO self_a VALUES (739 , 718 , 98 , 513 , true);
+INSERT INTO self_a VALUES (91 , 996 , 871 , 954 , true);
+INSERT INTO self_a VALUES (248 , 367 , 704 , 899 , true);
+INSERT INTO self_a VALUES (679 , 192 , 687 , 192 , true);
+INSERT INTO self_a VALUES (840 , 429 , 830 , 93 , false);
+INSERT INTO self_a VALUES (750 , 263 , 652 , 749 , true);
+INSERT INTO self_a VALUES (835 , 478 , 266 , 402 , true);
+INSERT INTO self_a VALUES (264 , 277 , 668 , 230 , true);
+INSERT INTO self_a VALUES (55 , 662 , 473 , 776 , true);
+INSERT INTO self_a VALUES (800 , 594 , 825 , 638 , true);
+INSERT INTO self_a VALUES (549 , 29 , 951 , 607 , false);
+INSERT INTO self_a VALUES (472 , 908 , 939 , 703 , true);
+INSERT INTO self_a VALUES (536 , 969 , 814 , 303 , true);
+INSERT INTO self_a VALUES (316 , 173 , 228 , 312 , false);
+INSERT INTO self_a VALUES (584 , 885 , 329 , 479 , false);
+INSERT INTO self_a VALUES (229 , 716 , 536 , 417 , false);
+INSERT INTO self_a VALUES (177 , 769 , 282 , 339 , true);
+INSERT INTO self_a VALUES (987 , 512 , 477 , 629 , false);
+INSERT INTO self_a VALUES (744 , 252 , 336 , 551 , false);
+INSERT INTO self_a VALUES (655 , 429 , 250 , 603 , true);
+INSERT INTO self_a VALUES (965 , 733 , 863 , 363 , false);
+INSERT INTO self_a VALUES (459 , 276 , 576 , 510 , true);
+INSERT INTO self_a VALUES (626 , 986 , 829 , 345 , false);
+INSERT INTO self_a VALUES (434 , 614 , 655 , 288 , true);
+INSERT INTO self_a VALUES (570 , 17 , 675 , 416 , true);
+INSERT INTO self_a VALUES (882 , 270 , 147 , 862 , false);
+INSERT INTO self_a VALUES (593 , 669 , 145 , 386 , false);
+INSERT INTO self_a VALUES (313 , 369 , 21 , 781 , true);
+INSERT INTO self_a VALUES (869 , 116 , 133 , 611 , true);
+INSERT INTO self_a VALUES (604 , 368 , 695 , 100 , true);
+INSERT INTO self_a VALUES (943 , 847 , 120 , 483 , true);
+INSERT INTO self_a VALUES (396 , 231 , 437 , 772 , false);
+INSERT INTO self_a VALUES (583 , 177 , 580 , 506 , true);
+INSERT INTO self_a VALUES (170 , 542 , 784 , 61 , false);
+INSERT INTO self_a VALUES (978 , 493 , 12 , 236 , false);
+INSERT INTO self_a VALUES (222 , 162 , 612 , 15 , false);
+INSERT INTO self_a VALUES (585 , 138 , 641 , 274 , true);
+INSERT INTO self_a VALUES (246 , 657 , 243 , 852 , true);
+INSERT INTO self_a VALUES (62 , 869 , 96 , 25 , false);
+INSERT INTO self_a VALUES (847 , 224 , 676 , 869 , false);
+INSERT INTO self_a VALUES (211 , 709 , 672 , 470 , true);
+INSERT INTO self_a VALUES (366 , 647 , 964 , 221 , false);
+INSERT INTO self_a VALUES (980 , 847 , 37 , 587 , false);
+INSERT INTO self_a VALUES (450 , 386 , 594 , 112 , false);
+INSERT INTO self_a VALUES (729 , 463 , 931 , 920 , false);
+INSERT INTO self_a VALUES (498 , 604 , 51 , 871 , false);
+INSERT INTO self_a VALUES (464 , 727 , 491 , 891 , false);
+INSERT INTO self_a VALUES (522 , 316 , 82 , 316 , true);
+INSERT INTO self_a VALUES (32 , 27 , 138 , 337 , false);
+INSERT INTO self_a VALUES (733 , 939 , 991 , 449 , true);
+INSERT INTO self_a VALUES (221 , 235 , 992 , 855 , false);
+INSERT INTO self_a VALUES (770 , 463 , 881 , 616 , false);
+INSERT INTO self_a VALUES (558 , 738 , 71 , 74 , false);
+INSERT INTO self_a VALUES (914 , 682 , 217 , 242 , false);
+INSERT INTO self_a VALUES (524 , 444 , 281 , 455 , false);
+INSERT INTO self_a VALUES (778 , 756 , 230 , 393 , false);
+INSERT INTO self_a VALUES (114 , 63 , 606 , 47 , true);
+INSERT INTO self_a VALUES (657 , 625 , 890 , 806 , false);
+INSERT INTO self_a VALUES (988 , 903 , 337 , 739 , true);
+INSERT INTO self_a VALUES (536 , 189 , 430 , 446 , true);
+INSERT INTO self_a VALUES (357 , 44 , 70 , 638 , false);
+INSERT INTO self_a VALUES (540 , 578 , 772 , 484 , true);
+INSERT INTO self_a VALUES (567 , 975 , 658 , 611 , true);
+INSERT INTO self_a VALUES (458 , 747 , 744 , 234 , false);
+INSERT INTO self_a VALUES (695 , 291 , 666 , 535 , false);
+INSERT INTO self_a VALUES (4 , 23 , 607 , 997 , true);
+INSERT INTO self_a VALUES (345 , 401 , 923 , 576 , true);
+INSERT INTO self_a VALUES (634 , 912 , 328 , 860 , true);
+INSERT INTO self_a VALUES (104 , 807 , 925 , 747 , true);
+INSERT INTO self_a VALUES (155 , 19 , 238 , 464 , true);
+INSERT INTO self_a VALUES (569 , 117 , 844 , 808 , false);
+INSERT INTO self_a VALUES (171 , 227 , 751 , 287 , true);
+INSERT INTO self_a VALUES (196 , 762 , 443 , 759 , true);
+INSERT INTO self_a VALUES (657 , 445 , 668 , 569 , false);
+INSERT INTO self_a VALUES (372 , 215 , 899 , 368 , true);
+INSERT INTO self_a VALUES (223 , 794 , 499 , 736 , false);
+INSERT INTO self_a VALUES (433 , 94 , 321 , 359 , true);
+INSERT INTO self_a VALUES (423 , 575 , 251 , 637 , false);
+INSERT INTO self_a VALUES (508 , 734 , 783 , 781 , true);
+INSERT INTO self_a VALUES (278 , 653 , 931 , 500 , false);
+INSERT INTO self_a VALUES (726 , 993 , 9 , 298 , true);
+INSERT INTO self_a VALUES (677 , 426 , 376 , 226 , false);
+INSERT INTO self_a VALUES (967 , 639 , 659 , 731 , false);
+INSERT INTO self_a VALUES (488 , 774 , 321 , 573 , false);
+INSERT INTO self_a VALUES (65 , 909 , 5 , 331 , true);
+INSERT INTO self_a VALUES (426 , 554 , 297 , 168 , true);
+INSERT INTO self_a VALUES (460 , 846 , 500 , 416 , true);
+INSERT INTO self_a VALUES (887 , 664 , 705 , 592 , true);
+INSERT INTO self_a VALUES (98 , 83 , 24 , 146 , false);
+INSERT INTO self_a VALUES (79 , 343 , 431 , 601 , false);
+INSERT INTO self_a VALUES (127 , 942 , 871 , 783 , true);
+INSERT INTO self_a VALUES (20 , 441 , 998 , 508 , false);
+INSERT INTO self_a VALUES (267 , 62 , 303 , 239 , true);
+INSERT INTO self_a VALUES (239 , 62 , 439 , 837 , true);
+INSERT INTO self_a VALUES (89 , 686 , 252 , 569 , false);
+INSERT INTO self_a VALUES (891 , 135 , 718 , 114 , true);
+INSERT INTO self_a VALUES (606 , 246 , 74 , 214 , false);
+INSERT INTO self_a VALUES (552 , 359 , 115 , 783 , false);
+INSERT INTO self_a VALUES (712 , 930 , 700 , 976 , false);
+INSERT INTO self_a VALUES (107 , 852 , 200 , 230 , true);
+INSERT INTO self_a VALUES (554 , 263 , 759 , 69 , false);
+INSERT INTO self_a VALUES (391 , 345 , 544 , 436 , false);
+INSERT INTO self_a VALUES (12 , 425 , 493 , 606 , false);
+INSERT INTO self_a VALUES (869 , 453 , 75 , 48 , true);
+INSERT INTO self_a VALUES (455 , 58 , 920 , 372 , true);
+INSERT INTO self_a VALUES (227 , 864 , 928 , 982 , false);
+INSERT INTO self_a VALUES (6 , 513 , 218 , 928 , false);
+INSERT INTO self_a VALUES (844 , 740 , 455 , 993 , false);
+INSERT INTO self_a VALUES (850 , 418 , 336 , 521 , false);
+INSERT INTO self_a VALUES (209 , 470 , 482 , 456 , true);
+INSERT INTO self_a VALUES (177 , 631 , 900 , 590 , true);
+INSERT INTO self_a VALUES (899 , 485 , 322 , 890 , true);
+INSERT INTO self_a VALUES (733 , 33 , 64 , 48 , true);
+INSERT INTO self_a VALUES (700 , 47 , 989 , 640 , false);
+INSERT INTO self_a VALUES (252 , 571 , 608 , 239 , false);
+INSERT INTO self_a VALUES (480 , 100 , 299 , 783 , false);
+INSERT INTO self_a VALUES (40 , 341 , 956 , 192 , false);
+INSERT INTO self_a VALUES (304 , 201 , 128 , 344 , true);
+INSERT INTO self_a VALUES (150 , 276 , 114 , 435 , false);
+INSERT INTO self_a VALUES (412 , 482 , 930 , 265 , false);
+INSERT INTO self_a VALUES (219 , 597 , 726 , 347 , true);
+INSERT INTO self_a VALUES (248 , 74 , 303 , 763 , true);
+INSERT INTO self_a VALUES (402 , 604 , 706 , 483 , true);
+INSERT INTO self_a VALUES (491 , 662 , 324 , 100 , false);
+INSERT INTO self_a VALUES (680 , 890 , 651 , 440 , true);
+INSERT INTO self_a VALUES (42 , 791 , 890 , 623 , true);
+INSERT INTO self_a VALUES (144 , 166 , 928 , 774 , false);
+INSERT INTO self_a VALUES (272 , 507 , 710 , 514 , true);
+
+INSERT INTO self_a VALUES (NULL, NULL, NULL, NULL);
+
+CREATE INDEX idx_self_a ON self_a USING BTREE (a , b , c);
+
+ANALYZE self_a;
+--
+-- Test single attribute self-contradictory
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a < 9;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 10 AND a < 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <= 10 AND a > 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <= 10 AND a >= 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 10 AND a < 10 AND a > 8 AND a <= 12;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a = 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 10 AND a = 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < 10 AND a = 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <= 10 AND a = 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = 10 AND a <> 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = 10 AND a <> 9;
+
+--
+-- Boolean type is a little tricky.
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS FALSE AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS FALSE AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS FALSE AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS UNKNOWN AND z IS TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS UNKNOWN AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS UNKNOWN AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT z AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= FALSE AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z >= TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z > TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z <= TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z < TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z < FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= TRUE AND z IS FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z >= FALSE AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS NOT FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z > FALSE AND z IS FALSE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z = true;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z > false;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z >= false;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> TRUE AND z >= true;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> FALSE AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> FALSE AND z >= false;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z <> FALSE AND z <= false;
+
+--
+-- Null test
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a IS NOT NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NOT NULL AND a > 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a = 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a > 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a <> 10;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z <> TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NULL AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND NOT z;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z <> TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS NOT TRUE;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS NOT UNKNOWN;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS NOT NULL AND z IS UNKNOWN;
+
+--
+-- Test weak restriction that have the sematic of same level will being eliminated and IS self contradictory or not
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a > 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a > 5 AND a < 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 5 AND a < 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 4 AND a >= 4 AND a < 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a >= 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a >= 5 AND a < 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a > 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a > 5 AND a < 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a = 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 4 AND a = 4 AND a <> 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 3 AND a = 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a >= 3 AND a = 4 AND a < 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 3 AND a = 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> 3 AND a = 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> 3 AND a = 4 AND a > 4;
+
+--
+-- Test same type coercion will be accepted
+--
+explain (costs off)
+select * from self_a WHERE a = 1 AND a <>'1'::int;
+
+--
+-- Test different type coercion will be ignored
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a = 1 AND a <>'1'::bigint;
+
+--
+-- Test specified collation will be ignored
+--
+explain (costs off)
+SELECT * FROM self_b WHERE a > 'c' AND a < 'b' COLLATE "pg_c_utf8";
+
+
+explain (costs off)
+SELECT * FROM self_b WHERE a > 'c' AND a < 'b';
+
+--
+-- Redundant clause will be eliminated
+--
+explain (costs off)
+select * from self_a WHERE a = 1 AND a = 1;
+
+explain (costs off)
+select * from self_a WHERE a > 10 AND a > 10;
+
+explain (costs off)
+select * from self_a WHERE a >= 10 AND a >= 10;
+
+explain (costs off)
+select * from self_a WHERE a < 4 AND a < 4;
+
+explain (costs off)
+select * from self_a WHERE a <= 4 AND a <= 4;
+
+--
+-- Test base or-clause
+--
+
+--
+-- Test or-clause self-contradictory
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a < 9 OR b > 8 AND b < 6;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND a < 9 OR ((b > 8 AND b < 7 or b = 10) AND b < 6);
+
+--
+-- Test or-clause can be pruned
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a < 9 AND a > 10 OR b < 8;
+
+SELECT * FROM self_a WHERE a < 9 AND a > 10 OR b < 8;
+
+--
+-- Test or-clause redundant clause will be eliminated
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < 14 AND a < 10 OR b < 8;
+
+SELECT * FROM self_a WHERE a < 14 AND a < 10 OR b < 8;
+
+--
+-- Test same level and-clause whether can be push down into or-clause to test self-contradictory and can be pruned
+--
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a < 9 OR b < 11);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a < 9 AND b < 11 OR b <=16);
+
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a < 9 AND b < 11 OR b <=16);
+
+--
+-- Test mixed expr clause
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE NOT(a = 1 AND a <>1) AND a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a IS NOT NULL OR a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS NULL AND a IS NOT NULL OR a = 1 AND a > 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z IS TRUE AND z IS NOT TRUE OR a = 1 AND a > 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE z = TRUE AND z IS NOT TRUE OR z IS NULL AND z IS NOT NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a BETWEEN 10 AND 5;
+
+explain (costs off)
+SELECT * FROM self_a WHERE (a > 4::bigint AND a > 4 AND a < 4 OR b > 3 AND b < 2) OR (a < 20);
+
+SELECT * FROM self_a WHERE (a > 1 AND a <1 OR b < 10 ) AND z IS TRUE OR b < 20 or z IS null;
+
+--
+-- Test pruned and self-contradictory mixed case
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE (a > 10 AND a > 12 OR b > 8) AND b < 7 AND b < 15;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (a > 10 AND a < 9 OR b < 11 OR a < 7);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > 10 AND b > 12 AND (((a < 9 OR b < 11) OR (b < 15 AND b < 17 OR a <12)) OR b >=8 AND a < 14);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> 3 AND a <> 4 AND a <> 8 AND (b > 12 AND c < 80 OR c = 16 AND (a = 4 OR a = 8));
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IS not null AND (b > 12 AND c < 80 OR c = 16 AND (a IS null OR a = 8));
+
+--
+-- Test self-contradictory join case
+--
+explain (costs off)
+SELECT * FROM self_a a LEFT JOIN self_a b ON a.a = b.a WHERE b.a > 300 AND a.a = 145;
+
+explain (costs off)
+select * from self_a a JOIN self_a b ON a.a =b.a WHERE a.a= 145 AND b.a < 79;
+
+--
+-- Test row comparison
+-- It's need to add define DEBUG_ROW_DECODE for debug
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b) > (NULL , 3);
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) < (100 , null , 3);
+
+--Equal expression
+--a<100
+--or a=100 and b<null
+--or a=100 and b=null and c<3
+
+--After converted
+--a<100
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) < (100 , null , 3) AND a = 101;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (1 , 2 , null);
+
+--Equal expression
+--a>1
+--or a=1 AND b>2
+--or a=1 AND b=2 AND c>null
+
+--After converted
+--a>1
+--or a=1 AND b>2
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (1 , 2 , null) AND a < 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(NULL , b) > (3 , 3);
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , null , b) > (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 and null>3
+--or a=2 and null=3 and b>4
+
+--After converted
+--a>2
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , null , b) > (2 , 3 , 4) AND a = 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , null) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 and b>3
+--or a=2 and b=3 and null>=4
+
+--After converted
+--a>2
+--or a=2 and b>3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , null) >= (2 , 3 , 4) AND a < 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , null) > (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 and b>3
+--or a=2 and b=3 and null>4
+
+--After converted
+--a>2
+--or a=2 and b>3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , 3 , 4);
+
+--Equal expression
+--a>a
+--or a=a AND b>3
+--or a=a AND b=3 AND c>=4
+
+--After converted
+--a is not null and b>3
+--or a IS not null and b=3 AND c>=4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , 3 , 4) AND a IS NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (a , 3 , 4);
+
+--Equal expression
+--a>a
+--or a=a AND b>3
+--or a=a AND b=3 AND c>4
+
+--After converted
+--a is not null and b>3
+--or a IS not null and b=3 AND c>4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , 4);
+
+--Equal expression
+--a>2
+--or a=2 and b>b
+--or a=2 AND b=b AND c >=4
+
+--After converted
+--a>2
+--or a=2 AND b IS not null AND c >=4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , 4) AND a = 2 AND c < 4;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , b , 4);
+
+--Equal expression
+--a>2
+--or a=2 and b>b
+--or a=2 AND b=b AND c >4
+
+--After converted
+--a>2
+--or a=2 AND b IS not null AND c >4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , 3 , c);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND c>=c
+
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND c IS not null
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , 3 , c) AND a = 2 AND c = 3 AND c IS NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , 3 , c);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND c>c
+
+--After converted
+--a>2
+--or a=2 AND b>3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , b , c);
+
+--Equal expression
+--a>2
+--or a=2 AND b>b
+--or a=2 AND b=b AND c>c
+
+--After converted
+--a>2
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (2 , b , c) AND a = 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , c);
+
+--Equal expression
+--a>2
+--or a=2 AND b>b
+--or a=2 AND b=b AND c>=c
+
+--After converted
+--a>2
+--or a=2 AND b=b AND c>=c
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (2 , b , c) AND a = 2 AND b IS NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) > (a , b , c);
+
+--Equal expression
+--a>a
+--or a=a AND b>b
+--or a=a AND b=b AND c>c
+
+--After converted
+--false
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , b , c);
+
+--Equal expression
+--a>a
+--or a=a AND b>b
+--or a=a AND b=b AND c>=c
+
+--After converted
+--a=a AND b=b AND c>=c
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , c) >= (a , b , c) AND a IS NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , b , c) >= (2 , 3 , 4);
+
+--Equal expression
+--2>2
+--or 2=2 AND b>3
+--or 2=2 AND b=3 AND c>=4
+
+--After converted
+--b>3
+--or b=3 AND c>=4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , b , c) >= (2 , 3 , 4) AND c < 4 AND b = 3;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , b , c) > (2 , 3 , 4);
+
+--Equal expression
+--2>2
+--or 2=2 AND b>3
+--or 2=2 AND b=3 AND c>=4
+
+--After converted
+--b>3
+--or b=3 AND c>4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , c) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND 3>3
+--or a=2 AND 3=3 AND c>=4
+
+--After converted
+--a>2
+--or a=2 AND c>=4
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , c) >= (2 , 3 , 4) AND c < 4 AND a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 4>=4
+
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) >= (2 , 3 , 4) AND b = 3 AND a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) > (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 4>4
+
+--After converted
+--a>2
+--or a=2 AND b>3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 4) > (2 , 3 , 4) AND a = 2 AND b = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND 3>3
+--or a=2 AND 3=3 AND 4>=4
+
+--After converted
+--a>2
+--or a=2
+
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (2 , 3 , 4) AND a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) > (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND 3>3
+--or a=2 AND 3=3 AND 4>4
+
+--After converted
+--a>2
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) > (2 , 3 , 4) AND a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , 3 , 4) >= (2 , 3 , 4);
+
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(2 , 3 , 4) > (2 , 3 , 4);
+
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) > (a , 3 , 4);
+
+--Equal expression
+--a>a
+--or a=a AND 3>3
+--or a=a AND 3=3 AND 4>4
+
+--After converted
+--false
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (a , 3 , 4);
+
+--Equal expression
+--a>a
+--or a=a AND 3>3
+--or a=a AND 3=3 AND 4>=4
+
+--After converted
+--a IS NOT NULL
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 3 , 4) >= (a , 3 , 4) AND a IS NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , b) > (4 , a , b);
+
+--After converted
+--false
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , b) >= (4 , a , b);
+
+--Equal expression
+--4>4
+--or 4=4 AND a>a
+--or 4=4 AND a=a AND b>=b
+
+--After converted
+--a IS NOT NULL and b IS NOT NULL
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , b) >= (4 , a , b) AND a IS NULL;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 4 , b) >= (a , 4 , b);
+
+--Equal expression
+--a>a
+--or a=a AND 4>4
+--or a=a AND 4=4 AND b>=b
+
+--After converted
+--a IS NOT NULL and b IS NOT NULL
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(4 , a , 4) >= (4 , a , 4);
+
+--Equal expression
+--4>4
+--or 4=4 AND a>a
+--or 4=4 AND a=a AND 4>=4
+
+--After converted
+--a IS NOT NULL
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , a , a) >= (a , a , a);
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , a , a) > (a , a , a);
+
+--Test const are not equal
+
+--negative
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(1 , b , c) >= (2 , 3 , 4);
+
+--Equal expression
+--1>2
+--or 1=2 AND b>3
+--or 1=2 AND b=3 AND c>=4
+
+--After converted
+--false
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(1 , b , c) > (2 , 3 , 4);
+
+--Equal expression
+--1>2
+--or 1=2 AND b>3
+--or 1=2 AND b=3 AND c>4
+
+--After converted
+--false
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , c) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND 2>3
+--or a=2 AND 2=3 AND c>=4
+
+--After converted
+--a>2
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , c) >= (2 , 3 , 4) AND a = 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 3>=4
+
+--After converted
+--a>2
+--or a=2 AND b>3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3) >= (2 , 3 , 4) AND a < 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3 , c) >= (2 , 3 , 4 , 5);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 3>=4
+--or a=2 AND b=3 AND 3=4 and c>=5
+
+--After converted
+--a>2
+--or a=2 AND b>3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 3) >= (a , b , 4);
+
+--Equal expression
+--a>a
+--or a=a AND b>b
+--or a=a AND b=b AND 3>=4
+
+--After converted
+--false
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 5 , b , 3) >= (a , 5 , b , 4);
+
+--Equal expression
+--a>a
+--or a=a AND 5>5
+--or a=a AND 5=5 and b>b
+--or a=a AND 5=5 and b=b and 3>=4
+
+--After converted
+--false
+
+--positive
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(5 , b , c) > (2 , 3 , 4);
+
+--Equal expression
+--5>2
+--or 5=2 AND b>3
+--or 5=2 AND b=3 AND c>4
+
+--After converted
+--true
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 5 , c) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND 5>3
+--or a=2 AND 5=3 AND c>=4
+
+--After converted
+--a>2
+--or a=2
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 5 , c) >= (2 , 3 , 4) AND a = 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 5) >= (2 , 3 , 4);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 5>=4
+
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 5) >= (2 , 3 , 4) AND a = 2 AND b < 3;
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b , 5 , c) >= (2 , 3 , 4 , 6);
+
+--Equal expression
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3 AND 5>=4
+--or a=2 AND b=3 AND 5=4 and c>=6
+
+--After converted
+--a>2
+--or a=2 AND b>3
+--or a=2 AND b=3
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , b , 4 , 3 , c) >= (108 , 2 , b , 2 , 4 , 5);
+
+--Equal expression
+--a>108
+--or a=108 and 2>2
+--or a=108 and 2=2 and b>b
+--or a=108 and 2=2 and b=b and 4>2
+--or a=108 and 2=2 and b=b and 4=2 and 3>4
+--or a=108 and 2=2 and b=b and 4=2 and 3=4 and c>=5
+
+--After converted
+--a>108
+--or a=108 and b IS NOT NULL
+
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , 2 , b , 4 , 3 , c) >= (108 , 2 , b , 2 , 4 , 5) AND a = 108 AND b is NULL;
+
+--Only keep expr that is simple after converted.
+explain (costs off)
+SELECT * FROM self_a WHERE ROW(a , b, c) > (23, 34, 56) AND b = 34 AND c = 56;
+
+
+--Test row comparison under the or-expr is const true and the or-expr will be const true.
+explain (costs off)
+SELECT * FROM self_a WHERE (ROW(2,a,b) > (1,3,5) OR c < 345 AND a = 89) AND a < 150 OR c = 178;
+
+--
+-- Test scalar array operation expr
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND a = 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND a < 20;
+
+SELECT COUNT(*) FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND a < 20;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 3 , 4]) AND (a = 2 or a = 4);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ALL (array[2 , 2]) AND a = 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a <> ANY (array[2 , 2]) AND a = 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ALL (array[20 , 30 , 40]) AND a = 27;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ALL (array[20 , 30 , 40]) AND a > 25;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ALL (array[20 , 20]) AND a >= 25;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[20 , 30 , 40]) AND a >= 25;
+
+SELECT COUNT(*) FROM self_a WHERE a < ANY (array[20 , 30 , 40]) AND a >= 25;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[20 , 30 , 40]) AND a > 40;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[20 , 20]) AND a > 20;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[2 , 3]);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[54, NULL]);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[2 , 2]) AND a <> 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ALL (array[27 , 27]) AND (a <> 27 OR a =27);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ANY (array[2]) AND a <> 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ANY (array[2 , 2]) AND a <> 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a = ANY (array[2 , 3 , 4]) AND a < 2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (2 , 3 , 4) AND a IN (5);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a < ANY (array[1 , 2]) AND a > ANY (array[2 , 3]);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ALL (array[2 , 3]) AND a <=3;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ANY (array[2 , 3]) AND a <=3;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ANY (array[2 , 3]) AND a <=2;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (58 , 32 , 4) AND a IN (1 , 32 , 6);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (58 , 32 , 4) AND a IN (1 , 3 , 6);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (1 , 2 , 3) AND a IN (1 , 3 , 6) AND a > 1;
+
+explain (costs off)
+SELECT * FROM self_a WHERE a > ANY (array[2 , 3]) AND a <=2 OR a = ANY (array[2 , 3]) AND a > 4;
+
+-- Test push scalar array expr down.
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (89, 12, 219) AND (b > 50 OR c < 300 AND a IN (89, 121, 319));
+
+explain (costs off)
+SELECT * FROM self_a WHERE a IN (89, 12, 219) AND (b > 50 OR (c < 345 AND c < 78 OR a IN (89, 121, 319)));
+
+--
+-- Test push scalar array expr that operator is <> down.
+--
+
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219) AND (b > 50 OR c < 345 AND a = 89);
+
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219) AND (b > 50 OR (c < 345 AND c < 78 OR a = 89));
+
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219) AND (b > 50 and a = 12 OR (c < 345 AND c < 78 and a = 219 OR a = 89));
+
+
+--Test using binary search.
+explain (costs off)
+SELECT * FROM self_a WHERE a not in (89, 12, 219, 78, 69) AND (b > 50 and a = 12 OR (c < 345 AND c < 78 and a = 219 OR a = 89));
\ No newline at end of file
--
2.43.0.windows.1
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: Self contradictory examining on rel's baserestrictinfo
@ 2024-11-25 20:55 Robert Haas <[email protected]>
parent: ro b <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Robert Haas @ 2024-11-25 20:55 UTC (permalink / raw)
To: ro b <[email protected]>; +Cc: pgsql-hackers <[email protected]>
On Mon, Nov 25, 2024 at 3:58 AM ro b <[email protected]> wrote:
> 1. Background
> A few months ago, when i read source codes of B-tree in routine
> _bt_preprocess_keys, i found that there are more contradictory
> checking case we can add. I sent email to pgsql-hackers and
> then community contributor replied me and told me someone had
> already proposed this question. Thanks for taking the time
> to address my question. After serveral conversations, i found
> that we can do something more. We can place these jobs at planning time.
When you're referring to things that happened in the past, you should
provide links to specific messages and names of specific contributors.
It will be difficult for anyone to find the previous discussion based
on your description of "a few months ago" and a "community
contributor".
I'm a little confused because it seems like you think we don't do any
of this kind of thing already. But:
robert.haas=# explain select * from foo where a < 1 and a > 1;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
robert.haas=# explain select * from foo where a < 1 and a = 1;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
robert.haas=# explain select * from foo where a <> 1 and a = 1;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
robert.haas=# explain select * from foo where a in (1,2,3) and a is null;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
There are cases where we don't already draw the necessary conclusions,
such as a>1 and a>2, which could be simplified to a>2. But those cases
aren't necessarily that common.
> 7) Scalar array comparison expression
> First we need to deconstruct the const array, figure out the null and non-null
> elements.
> If ALL flag is set and the Const contain NULL. we will get nothing (eg. x <=
> ALL(array[56, null])), it's contradictory.
True, but that already seems to be working:
robert.haas=# explain select * from foo where a <= all(array[56, null]);
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
I'm not saying there is no room for improvement here, but I think you
will need to (1) be more clear about exactly which cases we are
already handling vs. which ones you want to handle, (2) possibly split
the patch into smaller patches each of which handles one specific case
instead of bundling many improvements together, and (3) improve the
comments and commit message in the patch(es).
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: Self contradictory examining on rel's baserestrictinfo
@ 2024-11-27 14:30 ro b <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: ro b @ 2024-11-27 14:30 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers <[email protected]>
Thank you for your advice and guidance.
I didn't know the constraint_exclusion switch existed.
As your advice i need to make what i am doing to be clear.
> There are cases where we don't already draw the necessary conclusions,
> such as a>1 and a>2, which could be simplified to a>2. But those cases
> aren't necessarily that common.
The path i committed not just test contradictory but also do the
simplification. The simplification is limited in the BTREE.
Could you interpret the case in a little more detail.
Best regards
________________________________
From: Robert Haas <[email protected]>
Sent: Tuesday, November 26, 2024 04:55
To: ro b <[email protected]>
Cc: pgsql-hackers <[email protected]>
Subject: Re: Self contradictory examining on rel's baserestrictinfo
On Mon, Nov 25, 2024 at 3:58 AM ro b <[email protected]> wrote:
> 1. Background
> A few months ago, when i read source codes of B-tree in routine
> _bt_preprocess_keys, i found that there are more contradictory
> checking case we can add. I sent email to pgsql-hackers and
> then community contributor replied me and told me someone had
> already proposed this question. Thanks for taking the time
> to address my question. After serveral conversations, i found
> that we can do something more. We can place these jobs at planning time.
When you're referring to things that happened in the past, you should
provide links to specific messages and names of specific contributors.
It will be difficult for anyone to find the previous discussion based
on your description of "a few months ago" and a "community
contributor".
I'm a little confused because it seems like you think we don't do any
of this kind of thing already. But:
robert.haas=# explain select * from foo where a < 1 and a > 1;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
robert.haas=# explain select * from foo where a < 1 and a = 1;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
robert.haas=# explain select * from foo where a <> 1 and a = 1;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
robert.haas=# explain select * from foo where a in (1,2,3) and a is null;
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
There are cases where we don't already draw the necessary conclusions,
such as a>1 and a>2, which could be simplified to a>2. But those cases
aren't necessarily that common.
> 7) Scalar array comparison expression
> First we need to deconstruct the const array, figure out the null and non-null
> elements.
> If ALL flag is set and the Const contain NULL. we will get nothing (eg. x <=
> ALL(array[56, null])), it's contradictory.
True, but that already seems to be working:
robert.haas=# explain select * from foo where a <= all(array[56, null]);
QUERY PLAN
------------------------------------------
Result (cost=0.00..0.00 rows=0 width=0)
One-Time Filter: false
(2 rows)
I'm not saying there is no room for improvement here, but I think you
will need to (1) be more clear about exactly which cases we are
already handling vs. which ones you want to handle, (2) possibly split
the patch into smaller patches each of which handles one specific case
instead of bundling many improvements together, and (3) improve the
comments and commit message in the patch(es).
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: Self contradictory examining on rel's baserestrictinfo
@ 2024-11-27 14:52 Robert Haas <[email protected]>
parent: ro b <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: Robert Haas @ 2024-11-27 14:52 UTC (permalink / raw)
To: ro b <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers <[email protected]>
On Wed, Nov 27, 2024 at 9:30 AM ro b <[email protected]> wrote:
> The path i committed not just test contradictory but also do the
> simplification. The simplification is limited in the BTREE.
> Could you interpret the case in a little more detail.
I am not able to understand this paragraph.
Regretfully,
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: Self contradictory examining on rel's baserestrictinfo
@ 2024-11-27 15:28 ro b <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 84+ messages in thread
From: ro b @ 2024-11-27 15:28 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers <[email protected]>
I mean why can't draw the conclusions
like the case a>1 and a>2 which can be simplified
to a>2 if the operator > is btree opclass.
Best regards
________________________________
From: Robert Haas <[email protected]>
Sent: Wednesday, November 27, 2024 22:52
To: ro b <[email protected]>
Cc: Peter Geoghegan <[email protected]>; pgsql-hackers <[email protected]>
Subject: Re: Self contradictory examining on rel's baserestrictinfo
On Wed, Nov 27, 2024 at 9:30 AM ro b <[email protected]> wrote:
> The path i committed not just test contradictory but also do the
> simplification. The simplification is limited in the BTREE.
> Could you interpret the case in a little more detail.
I am not able to understand this paragraph.
Regretfully,
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 84+ messages in thread
* Re: Self contradictory examining on rel's baserestrictinfo
@ 2024-11-27 16:09 Robert Haas <[email protected]>
parent: ro b <[email protected]>
0 siblings, 0 replies; 84+ messages in thread
From: Robert Haas @ 2024-11-27 16:09 UTC (permalink / raw)
To: ro b <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers <[email protected]>
On Wed, Nov 27, 2024 at 10:28 AM ro b <[email protected]> wrote:
> I mean why can't draw the conclusions
> like the case a>1 and a>2 which can be simplified
> to a>2 if the operator > is btree opclass.
This question has already been discussed in some detail on this email
thread. First, we sometimes do detect it, as discussed by Peter.
Second, it is an unusual case and so not worth spending a lot of CPU
cycles to detect, as discussed by Tom.
We might still accept a patch to improve things in this area. For
example, if somebody could find a way to change the code so that we
are able to detect more cases without needing to spend more CPU
cycles, I think everyone would like that. However, that may be a
tricky goal to accomplish.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 84+ messages in thread
end of thread, other threads:[~2024-11-27 16:09 UTC | newest]
Thread overview: 84+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-01-04 09:45 [PATCH v21 4/7] Add support for PRESERVE dilipkumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH 08/12] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 7/7] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-02-10 06:19 [PATCH v24 07/10] Add support for PRESERVE Dilip Kumar <[email protected]>
2021-12-18 01:53 sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-18 04:52 ` Re: sequences vs. synchronous replication Tom Lane <[email protected]>
2021-12-18 06:00 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-18 20:45 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-18 21:27 ` Re: sequences vs. synchronous replication Tom Lane <[email protected]>
2021-12-18 21:48 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-18 21:51 ` Re: sequences vs. synchronous replication Tom Lane <[email protected]>
2021-12-20 14:31 ` Re: sequences vs. synchronous replication Peter Eisentraut <[email protected]>
2021-12-20 16:40 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-21 00:53 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-21 01:01 ` Re: sequences vs. synchronous replication Tom Lane <[email protected]>
2021-12-21 02:49 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-22 18:49 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-23 14:42 ` Re: sequences vs. synchronous replication Fujii Masao <[email protected]>
2021-12-23 18:50 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-24 05:37 ` Re: sequences vs. synchronous replication Kyotaro Horiguchi <[email protected]>
2021-12-24 07:23 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-24 08:04 ` Re: sequences vs. synchronous replication Kyotaro Horiguchi <[email protected]>
2021-12-24 10:40 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-28 06:56 ` Re: sequences vs. synchronous replication Fujii Masao <[email protected]>
2021-12-28 08:28 ` Re: sequences vs. synchronous replication Sascha Kuhl <[email protected]>
2021-12-28 08:50 ` Re: sequences vs. synchronous replication Pavel Stehule <[email protected]>
2021-12-28 08:53 ` Re: sequences vs. synchronous replication Sascha Kuhl <[email protected]>
2021-12-28 08:57 ` Re: sequences vs. synchronous replication Pavel Stehule <[email protected]>
2021-12-27 20:24 ` Re: sequences vs. synchronous replication Peter Eisentraut <[email protected]>
2021-12-28 01:39 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-19 03:03 ` Re: sequences vs. synchronous replication Amit Kapila <[email protected]>
2021-12-22 01:57 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-22 04:56 ` Re: sequences vs. synchronous replication Fujii Masao <[email protected]>
2021-12-22 12:11 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2021-12-22 17:50 ` Re: sequences vs. synchronous replication Fujii Masao <[email protected]>
2021-12-22 19:00 ` Re: sequences vs. synchronous replication Tomas Vondra <[email protected]>
2024-11-25 08:58 Self contradictory examining on rel's baserestrictinfo ro b <[email protected]>
2024-11-25 20:55 ` Re: Self contradictory examining on rel's baserestrictinfo Robert Haas <[email protected]>
2024-11-27 14:30 ` Re: Self contradictory examining on rel's baserestrictinfo ro b <[email protected]>
2024-11-27 14:52 ` Re: Self contradictory examining on rel's baserestrictinfo Robert Haas <[email protected]>
2024-11-27 15:28 ` Re: Self contradictory examining on rel's baserestrictinfo ro b <[email protected]>
2024-11-27 16:09 ` Re: Self contradictory examining on rel's baserestrictinfo Robert Haas <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox