agora inbox for [email protected]
help / color / mirror / Atom feed[PATCH v21 5/7] Create custom compression methods
26+ messages / 3 participants
[nested] [flat]
* [PATCH v21 5/7] Create custom compression methods
@ 2021-01-29 10:14 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-01-29 10:14 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 7 ++-
doc/src/sgml/ref/create_access_method.sgml | 13 ++--
doc/src/sgml/ref/create_table.sgml | 3 +-
src/backend/access/common/detoast.c | 59 +++++++++++++++++--
src/backend/access/common/toast_internals.c | 19 +++++-
src/backend/access/compression/compress_lz4.c | 21 +++----
.../access/compression/compress_pglz.c | 20 +++----
.../access/compression/compressamapi.c | 27 +++++++++
src/backend/access/index/amapi.c | 50 ++++++++++++----
src/backend/commands/amcmds.c | 5 ++
src/backend/executor/nodeModifyTable.c | 2 +-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/varlena.c | 7 +--
src/bin/pg_dump/pg_dump.c | 3 +
src/bin/psql/tab-complete.c | 6 ++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 16 +++--
src/include/access/detoast.h | 8 +++
src/include/access/toast_internals.h | 15 +++++
src/test/regress/expected/compression.out | 40 ++++++++++++-
src/test/regress/expected/compression_1.out | 41 ++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
22 files changed, 312 insertions(+), 62 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 081fe078a4..46f34bffe5 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -390,9 +390,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This clause adds compression to a column. The Compression method can be
- set to any available compression method. The built-in methods are
- <literal>pglz</literal> and <literal>lz4</literal>.
+ This clause adds compression to a column. The Compression method could be
+ created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The built-in methods
+ are <literal>pglz</literal> and <literal>lz4</literal>.
The <literal>PRESERVE</literal> list contains a list of compression
methods used in the column and determines which of them may be kept.
Without <literal>PRESERVE</literal> or if any of the pre-existing
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..79f1290a58 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,7 +61,7 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
+ <literal>TABLE</literal>, <literal>INDEX</literal> and <literal>COMPRESSION</literal>
are supported at present.
</para>
</listitem>
@@ -77,12 +77,15 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
- API is described in <xref linkend="indexam"/>.
+ is described in <xref linkend="tableam"/>, the index access method
+ API is described in <xref linkend="indexam"/> an the compression access
+ method API is described in <xref linkend="compression-am"/>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index d514c7f8c1..10b468fb2d 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -998,7 +998,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<listitem>
<para>
This clause adds the compression method to a column. The Compression
- method can be set from available compression methods. The built-in
+ method could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set from the available compression methods. The built-in
methods are <literal>pglz</literal> and <literal>lz4</literal>.
If no compression method is specified, then compressible types will have
the default compression method <literal>pglz</literal>.
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index c90464053c..e032c30d25 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -457,13 +457,44 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
return result;
}
+/* ----------
+ * toast_get_compression_oid -
+ *
+ * Return the Oid of the compresion method stored in the compressed data.
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
+ */
+Oid
+toast_get_compression_oid(struct varlena *attr)
+{
+ CompressionId cmid;
+
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
+}
+
/* ----------
* toast_get_compression_handler - get the compression handler routines
*
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -477,10 +508,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "Invalid compression method id %d", cmid);
}
@@ -496,9 +538,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -512,16 +556,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index e8b035d138..34b0dc1002 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -30,7 +30,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -45,10 +45,10 @@ lz4_cmcompress(const struct varlena *value)
valsize = VARSIZE_ANY_EXHDR(value);
max_size = LZ4_compressBound(VARSIZE_ANY_EXHDR(value));
- tmp = (struct varlena *) palloc(max_size + TOAST_COMPRESS_HDRSZ);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + TOAST_COMPRESS_HDRSZ,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
{
@@ -56,7 +56,7 @@ lz4_cmcompress(const struct varlena *value)
elog(ERROR, "lz4: could not compress data");
}
- SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -68,7 +68,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -81,9 +81,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ);
SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ);
- rawsize = LZ4_decompress_safe(TOAST_COMPRESS_RAWDATA(value),
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - TOAST_COMPRESS_HDRSZ,
+ VARSIZE(value) - header_size,
TOAST_COMPRESS_RAWSIZE(value));
if (rawsize < 0)
elog(ERROR, "lz4: compressed data is corrupted");
@@ -100,7 +100,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -112,9 +113,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ);
- rawsize = LZ4_decompress_safe_partial(TOAST_COMPRESS_RAWDATA(value),
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - TOAST_COMPRESS_HDRSZ,
+ VARSIZE(value) - header_size,
slicelength,
TOAST_COMPRESS_RAWSIZE(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 578be6f1dd..d33ea3d2cf 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -44,16 +44,16 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- TOAST_COMPRESS_HDRSZ);
+ header_size);
len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)),
valsize,
- TOAST_COMPRESS_RAWDATA(tmp),
+ (char *) tmp + header_size,
NULL);
if (len >= 0)
{
- SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -68,7 +68,7 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
@@ -76,8 +76,8 @@ pglz_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ);
SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ);
- rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value),
- TOAST_COMPRESS_SIZE(value),
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
TOAST_COMPRESS_RAWSIZE(value), true);
@@ -95,7 +95,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -103,8 +103,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value),
- VARSIZE(value) - TOAST_COMPRESS_HDRSZ,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c
index e8bfb23ec3..19780f57e7 100644
--- a/src/backend/access/compression/compressamapi.c
+++ b/src/backend/access/compression/compressamapi.c
@@ -19,6 +19,7 @@
#include "access/htup_details.h"
#include "access/reloptions.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "commands/defrem.h"
#include "utils/fmgroids.h"
#include "utils/syscache.h"
@@ -63,6 +64,32 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
/* check_hook: validate new default_toast_compression */
bool
check_default_toast_compression(char **newval, void **extra, GucSource source)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..89a4b01d78 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -46,14 +46,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +64,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +93,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 668e43faed..31d4a3e958 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -241,6 +241,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "TABLE";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -278,6 +280,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 916f6541c1..a63737213d 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2089,7 +2089,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
* compression method is not supported by the target attribute then
* we need to decompress it.
*/
- cmoid = CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value));
+ cmoid = toast_get_compression_oid(new_value);
if (!IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 332e60f170..0a0ea0562a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5304,6 +5304,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index aff5b7e95e..ebd797158a 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -5309,7 +5309,6 @@ Datum
pg_column_compression(PG_FUNCTION_ARGS)
{
Datum value = PG_GETARG_DATUM(0);
- char *compression;
int typlen;
struct varlena *varvalue;
@@ -5339,10 +5338,8 @@ pg_column_compression(PG_FUNCTION_ARGS)
varvalue = (struct varlena *) DatumGetPointer(value);
- compression =
- get_am_name(CompressionIdToOid(TOAST_COMPRESS_METHOD(varvalue)));
-
- PG_RETURN_TEXT_P(cstring_to_text(compression));
+ PG_RETURN_TEXT_P(cstring_to_text(get_am_name(
+ toast_get_compression_oid(varvalue))));
}
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c09f3972c6..3a79754550 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13124,6 +13124,9 @@ dumpAccessMethod(Archive *fout, AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index c40d498a30..5de2a6a18b 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..0f7b5b0794 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
#endif /* AMAPI_H */
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index be5b4da7b8..36c9f57628 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -30,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
#define DEFAULT_TOAST_COMPRESSION "pglz"
@@ -38,13 +40,18 @@ typedef enum CompressionId
extern char *default_toast_compression;
/* Use default compression method if it is not specified. */
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -68,6 +75,7 @@ extern const CompressionAmRoutine lz4_compress_methods;
/* access/compression/compressamapi.c */
extern CompressionId CompressionOidToId(Oid cmoid);
extern Oid CompressionIdToOid(CompressionId cmid);
+extern CompressionAmRoutine *GetCompressionAmRoutineByAmId(Oid amoid);
extern void assign_default_toast_compression(const char *newval, void **extra);
extern bool check_default_toast_compression(char **newval, void **extra, GucSource source);
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 0adf53c77b..6cdc37542d 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value);
*/
extern Size toast_datum_size(Datum value);
+/* ----------
+ * toast_get_compression_oid -
+ *
+ * Return the compression method oid from the compressed value
+ * ----------
+ */
+extern Oid toast_get_compression_oid(struct varlena *attr);
+
#endif /* DETOAST_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 84ba303e7e..c158354272 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,6 +27,17 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
#define RAWSIZEMASK (0x3FFFFFFFU)
/*
@@ -38,6 +49,7 @@ typedef struct toast_compress_header
* two highest bits.
*/
#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->info & RAWSIZEMASK)
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> 30)
#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ)
@@ -49,6 +61,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << 30); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 151478bcdd..636ab4021b 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -235,13 +235,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 81182fe949..3177ff91e8 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -232,13 +232,52 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index e447316573..0552eeca16 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -95,6 +95,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--ytoMbUMiTKPMT3hY
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v21-0006-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH 09/12] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/expected/compression_1.out | 41 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
22 files changed, 344 insertions(+), 59 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 807597e648..5e13b3288c 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 05c782e8e9..fe4be7bda4 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4b3d34ae1a..6556fe847e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 36a5f8ba5e..d4e215c931 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -241,13 +241,52 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--9Ek0hoCL9XbhcSqy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0010-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v24 08/10] Create custom compression methods
@ 2021-02-10 10:57 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Dilip Kumar @ 2021-02-10 10:57 UTC (permalink / raw)
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
XXX: this is failing pg_dump tests
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
21 files changed, 304 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index a4b297340f..196352ef42 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 869fd3676d..8ca2b832a6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4a13844ce6..64c9c6e538 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
--m51xatjYGsM+13rf
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0009-new-compression-method-extension-for-zlib.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v13 1/2] Key PGSTAT_KIND_RELATION by relfile locator
@ 2025-10-01 09:45 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Bertrand Drouvot @ 2025-10-01 09:45 UTC (permalink / raw)
This patch changes the key used for the PGSTAT_KIND_RELATION statistic kind.
Instead of the relation oid, it now relies on:
- dboid (linked to RelFileLocator's dbOid)
- objoid which is the result of a new macro (namely RelFileLocatorToPgStatObjid())
that computes an objoid based on the RelFileLocator's spcOid and the
RelFileLocator's relNumber.
This is possible as, since b14e9ce7d55c, the objoid is now uint64 and spcOid
and relNumber are 32 bits.
That will allow us to add new stats (add writes counters) and ensure that some
counters (n_dead_tup and friends) are replicated.
The patch introduces pgstat_reloid_to_relfilelocator() to 1) avoid calling
RelationIdGetRelation() to get the relfilelocator based on the relation oid
and 2) handle the partitioned table case.
Please note that:
- when running pg_stat_have_stats('relation',...) we now need to be connected
to the database that hosts the relation. As pg_stat_have_stats() is not
documented publicly, then the changes done in 029_stats_restart.pl look
enough.
- this patch does not handle rewrites so some tests are failing. It's only
intent is to ease the review and should not be pushed without being
merged with the following patch that handles the rewrites.
- it can be used to test that stats are incremented correctly and that we're
able to retrieve them as long as rewrites are not involved.
---
src/backend/postmaster/autovacuum.c | 17 +-
src/backend/utils/activity/pgstat_relation.c | 236 ++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 21 +-
src/include/catalog/pg_tablespace.dat | 4 +
src/include/catalog/pg_tablespace.h | 8 +
src/include/pgstat.h | 15 +-
src/include/utils/pgstat_internal.h | 1 +
src/test/recovery/t/029_stats_restart.pl | 40 ++--
8 files changed, 271 insertions(+), 71 deletions(-)
6.1% src/backend/postmaster/
61.7% src/backend/utils/activity/
4.9% src/backend/utils/adt/
3.2% src/include/catalog/
5.6% src/include/
18.1% src/test/recovery/t/
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 219673db930..4bef7ab7543 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -1992,12 +1992,16 @@ do_autovacuum(void)
bool dovacuum;
bool doanalyze;
bool wraparound;
+ RelFileLocator locator;
if (classForm->relkind != RELKIND_RELATION &&
classForm->relkind != RELKIND_MATVIEW)
continue;
relid = classForm->oid;
+ locator.dbOid = classForm->relisshared ? InvalidOid : MyDatabaseId;
+ locator.spcOid = classForm->reltablespace;
+ locator.relNumber = classForm->relfilenode;
/*
* Check if it is a temp table (presumably, of some other backend's).
@@ -2026,8 +2030,7 @@ do_autovacuum(void)
/* Fetch reloptions and the pgstat entry for this table */
relopts = extract_autovac_opts(tuple, pg_class_desc);
- tabentry = pgstat_fetch_stat_tabentry_ext(classForm->relisshared,
- relid);
+ tabentry = pgstat_fetch_stat_tabentry_by_locator(locator);
/* Check if it needs vacuum or analyze */
relation_needs_vacanalyze(relid, relopts, classForm, tabentry,
@@ -2092,6 +2095,7 @@ do_autovacuum(void)
bool dovacuum;
bool doanalyze;
bool wraparound;
+ RelFileLocator locator;
/*
* We cannot safely process other backends' temp tables, so skip 'em.
@@ -2100,6 +2104,9 @@ do_autovacuum(void)
continue;
relid = classForm->oid;
+ locator.dbOid = classForm->relisshared ? InvalidOid : MyDatabaseId;
+ locator.spcOid = classForm->reltablespace;
+ locator.relNumber = classForm->relfilenode;
/*
* fetch reloptions -- if this toast table does not have them, try the
@@ -2119,8 +2126,7 @@ do_autovacuum(void)
}
/* Fetch the pgstat entry for this table */
- tabentry = pgstat_fetch_stat_tabentry_ext(classForm->relisshared,
- relid);
+ tabentry = pgstat_fetch_stat_tabentry_by_locator(locator);
relation_needs_vacanalyze(relid, relopts, classForm, tabentry,
effective_multixact_freeze_max_age,
@@ -2917,8 +2923,7 @@ recheck_relation_needs_vacanalyze(Oid relid,
PgStat_StatTabEntry *tabentry;
/* fetch the pgstat table entry */
- tabentry = pgstat_fetch_stat_tabentry_ext(classForm->relisshared,
- relid);
+ tabentry = pgstat_fetch_stat_tabentry_ext(relid);
relation_needs_vacanalyze(relid, avopts, classForm, tabentry,
effective_multixact_freeze_max_age,
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..89bf0cbed56 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -17,12 +17,17 @@
#include "postgres.h"
+#include "access/htup_details.h"
#include "access/twophase_rmgr.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "catalog/pg_tablespace.h"
+#include "storage/lmgr.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
#include "utils/rel.h"
+#include "utils/relmapper.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
@@ -36,13 +41,12 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter inserted_pre_truncdrop;
PgStat_Counter updated_pre_truncdrop;
PgStat_Counter deleted_pre_truncdrop;
- Oid id; /* table's OID */
- bool shared; /* is it a shared catalog? */
+ RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
} TwoPhasePgStatRecord;
-static PgStat_TableStatus *pgstat_prep_relation_pending(Oid rel_id, bool isshared);
+static PgStat_TableStatus *pgstat_prep_relation_pending(RelFileLocator locator);
static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_level);
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
@@ -60,8 +64,7 @@ pgstat_copy_relation_stats(Relation dst, Relation src)
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(src->rd_rel->relisshared,
- RelationGetRelid(src));
+ srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
if (!srcstats)
return;
@@ -94,8 +97,10 @@ pgstat_init_relation(Relation rel)
/*
* We only count stats for relations with storage and partitioned tables
+ * and we don't count stats generated during a rewrite.
*/
- if (!RELKIND_HAS_STORAGE(relkind) && relkind != RELKIND_PARTITIONED_TABLE)
+ if ((!RELKIND_HAS_STORAGE(relkind) && relkind != RELKIND_PARTITIONED_TABLE) ||
+ OidIsValid(rel->rd_rel->relrewrite))
{
rel->pgstat_enabled = false;
rel->pgstat_info = NULL;
@@ -130,12 +135,37 @@ pgstat_init_relation(Relation rel)
void
pgstat_assoc_relation(Relation rel)
{
+ RelFileLocator locator;
+
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
+ /*
+ * Don't associate stats for relations without storage and non partitioned
+ * tables.
+ */
+ if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind) &&
+ rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
+ return;
+
+ if (rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
+ locator = rel->rd_locator;
+ else
+ {
+ /*
+ * Partitioned tables don't have storage, so construct a synthetic
+ * locator for statistics tracking. Use a reserved pseudo tablespace
+ * OID that cannot conflict with real tablespaces, and the relation
+ * OID as relNumber. This ensures no collision with regular relations
+ * even after OID wraparound.
+ */
+ locator.dbOid = (rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId);
+ locator.spcOid = PSEUDO_PARTITION_TABLE_SPCOID;
+ locator.relNumber = rel->rd_id;
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(RelationGetRelid(rel),
- rel->rd_rel->relisshared);
+ rel->pgstat_info = pgstat_prep_relation_pending(locator);
/* don't allow link a stats to multiple relcache entries */
Assert(rel->pgstat_info->relation == NULL);
@@ -167,9 +197,13 @@ pgstat_unlink_relation(Relation rel)
void
pgstat_create_relation(Relation rel)
{
+ /* don't track stats for relations without storage */
+ if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ return;
+
pgstat_create_transactional(PGSTAT_KIND_RELATION,
- rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(rel));
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
}
/*
@@ -181,9 +215,13 @@ pgstat_drop_relation(Relation rel)
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ /* don't track stats for relations without storage */
+ if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ return;
+
pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(rel));
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -213,20 +251,23 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
PgStat_EntryRef *entry_ref;
PgStatShared_Relation *shtabentry;
PgStat_StatTabEntry *tabentry;
- Oid dboid = (rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId);
TimestampTz ts;
PgStat_Counter elapsedtime;
+ RelFileLocator locator;
if (!pgstat_track_counts)
return;
+ locator = rel->rd_locator;
+
/* Store the data in the table's hash table entry. */
ts = GetCurrentTimestamp();
elapsedtime = TimestampDifferenceMilliseconds(starttime, ts);
/* block acquiring lock for the same reason as pgstat_report_autovac() */
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, dboid,
- RelationGetRelid(rel), false);
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, locator.dbOid,
+ RelFileLocatorToPgStatObjid(locator),
+ false);
shtabentry = (PgStatShared_Relation *) entry_ref->shared_stats;
tabentry = &shtabentry->stats;
@@ -285,9 +326,9 @@ pgstat_report_analyze(Relation rel,
PgStat_EntryRef *entry_ref;
PgStatShared_Relation *shtabentry;
PgStat_StatTabEntry *tabentry;
- Oid dboid = (rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId);
TimestampTz ts;
PgStat_Counter elapsedtime;
+ RelFileLocator locator;
if (!pgstat_track_counts)
return;
@@ -325,9 +366,25 @@ pgstat_report_analyze(Relation rel,
ts = GetCurrentTimestamp();
elapsedtime = TimestampDifferenceMilliseconds(starttime, ts);
+ if (rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
+ locator = rel->rd_locator;
+ else
+ {
+ /*
+ * Partitioned tables don't have storage, so construct a synthetic
+ * locator for statistics tracking. Use a reserved pseudo tablespace
+ * OID that cannot conflict with real tablespaces, and the relation
+ * OID as relNumber. This ensures no collision with regular relations
+ * even after OID wraparound.
+ */
+ locator.dbOid = (rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId);
+ locator.spcOid = PSEUDO_PARTITION_TABLE_SPCOID;
+ locator.relNumber = rel->rd_id;
+ }
/* block acquiring lock for the same reason as pgstat_report_autovac() */
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, dboid,
- RelationGetRelid(rel),
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
+ locator.dbOid,
+ RelFileLocatorToPgStatObjid(locator),
false);
/* can't get dropped while accessed */
Assert(entry_ref != NULL && entry_ref->shared_stats != NULL);
@@ -468,7 +525,16 @@ pgstat_update_heap_dead_tuples(Relation rel, int delta)
PgStat_StatTabEntry *
pgstat_fetch_stat_tabentry(Oid relid)
{
- return pgstat_fetch_stat_tabentry_ext(IsSharedRelation(relid), relid);
+ return pgstat_fetch_stat_tabentry_ext(relid);
+}
+
+PgStat_StatTabEntry *
+pgstat_fetch_stat_tabentry_by_locator(RelFileLocator locator)
+{
+ return (PgStat_StatTabEntry *) pgstat_fetch_entry(
+ PGSTAT_KIND_RELATION,
+ locator.dbOid,
+ RelFileLocatorToPgStatObjid(locator));
}
/*
@@ -476,12 +542,14 @@ pgstat_fetch_stat_tabentry(Oid relid)
* whether the to-be-accessed table is a shared relation or not.
*/
PgStat_StatTabEntry *
-pgstat_fetch_stat_tabentry_ext(bool shared, Oid reloid)
+pgstat_fetch_stat_tabentry_ext(Oid reloid)
{
- Oid dboid = (shared ? InvalidOid : MyDatabaseId);
+ RelFileLocator locator;
- return (PgStat_StatTabEntry *)
- pgstat_fetch_entry(PGSTAT_KIND_RELATION, dboid, reloid);
+ if (!pgstat_reloid_to_relfilelocator(reloid, &locator))
+ return NULL;
+
+ return pgstat_fetch_stat_tabentry_by_locator(locator);
}
/*
@@ -503,14 +571,17 @@ find_tabstat_entry(Oid rel_id)
PgStat_TableXactStatus *trans;
PgStat_TableStatus *tabentry = NULL;
PgStat_TableStatus *tablestatus = NULL;
+ RelFileLocator locator;
+
+ if (!pgstat_reloid_to_relfilelocator(rel_id, &locator))
+ return NULL;
+
+ entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ locator.dbOid,
+ RelFileLocatorToPgStatObjid(locator));
- entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION, MyDatabaseId, rel_id);
if (!entry_ref)
- {
- entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION, InvalidOid, rel_id);
- if (!entry_ref)
- return tablestatus;
- }
+ return tablestatus;
tabentry = (PgStat_TableStatus *) entry_ref->pending;
tablestatus = palloc_object(PgStat_TableStatus);
@@ -706,8 +777,12 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.inserted_pre_truncdrop = trans->inserted_pre_truncdrop;
record.updated_pre_truncdrop = trans->updated_pre_truncdrop;
record.deleted_pre_truncdrop = trans->deleted_pre_truncdrop;
- record.id = tabstat->id;
- record.shared = tabstat->shared;
+
+ if (tabstat->relation != NULL)
+ record.locator = tabstat->relation->rd_locator;
+ else
+ record.locator = tabstat->locator;
+
record.truncdropped = trans->truncdropped;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
@@ -750,7 +825,7 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
PgStat_TableStatus *pgstat_info;
/* Find or create a tabstat entry for the rel */
- pgstat_info = pgstat_prep_relation_pending(rec->id, rec->shared);
+ pgstat_info = pgstat_prep_relation_pending(rec->locator);
/* Same math as in AtEOXact_PgStat, commit case */
pgstat_info->counts.tuples_inserted += rec->tuples_inserted;
@@ -785,8 +860,8 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
- /* Find or create a tabstat entry for the rel */
- pgstat_info = pgstat_prep_relation_pending(rec->id, rec->shared);
+ /* Find or create a tabstat entry for the target locator */
+ pgstat_info = pgstat_prep_relation_pending(rec->locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -920,17 +995,21 @@ pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts)
* initialized if not exists.
*/
static PgStat_TableStatus *
-pgstat_prep_relation_pending(Oid rel_id, bool isshared)
+pgstat_prep_relation_pending(RelFileLocator locator)
{
PgStat_EntryRef *entry_ref;
PgStat_TableStatus *pending;
+ uint64 objid;
+
+ objid = RelFileLocatorToPgStatObjid(locator);
entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_RELATION,
- isshared ? InvalidOid : MyDatabaseId,
- rel_id, NULL);
+ locator.dbOid,
+ objid, NULL);
+
pending = entry_ref->pending;
- pending->id = rel_id;
- pending->shared = isshared;
+ pending->id = objid;
+ pending->locator = locator;
return pending;
}
@@ -1009,3 +1088,82 @@ restore_truncdrop_counters(PgStat_TableXactStatus *trans)
trans->tuples_deleted = trans->deleted_pre_truncdrop;
}
}
+
+/*
+ * Convert a relation OID to its corresponding RelFileLocator for statistics
+ * tracking purposes.
+ *
+ * Returns true on success, false if the relation doesn't need statistics
+ * tracking.
+ *
+ * For partitioned tables, constructs a synthetic locator using the relation
+ * OID as relNumber, since they don't have storage.
+ */
+bool
+pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
+{
+ HeapTuple tuple;
+ Form_pg_class relform;
+ bool result = true;
+
+ /* get the relation's tuple from pg_class */
+ tuple = SearchSysCache1(RELOID, ObjectIdGetDatum(reloid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ relform = (Form_pg_class) GETSTRUCT(tuple);
+
+ /* skip relations without storage and non partitioned tables */
+ if (!RELKIND_HAS_STORAGE(relform->relkind) &&
+ relform->relkind != RELKIND_PARTITIONED_TABLE)
+ {
+ ReleaseSysCache(tuple);
+ return false;
+ }
+
+ if (relform->relkind != RELKIND_PARTITIONED_TABLE)
+ {
+ /* build the RelFileLocator */
+ locator->relNumber = relform->relfilenode;
+ locator->spcOid = relform->reltablespace;
+
+ /* handle default tablespace */
+ if (!OidIsValid(locator->spcOid))
+ locator->spcOid = MyDatabaseTableSpace;
+
+ /* handle dbOid for global vs local relations */
+ if (locator->spcOid == GLOBALTABLESPACE_OID)
+ locator->dbOid = InvalidOid;
+ else
+ locator->dbOid = MyDatabaseId;
+
+ /* handle mapped relations */
+ if (!RelFileNumberIsValid(locator->relNumber))
+ {
+ locator->relNumber = RelationMapOidToFilenumber(reloid,
+ relform->relisshared);
+ if (!RelFileNumberIsValid(locator->relNumber))
+ {
+ ReleaseSysCache(tuple);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ /*
+ * Partitioned tables don't have storage, so construct a synthetic
+ * locator for statistics tracking. Use a reserved pseudo tablespace
+ * OID that cannot conflict with real tablespaces, and the relation
+ * OID as relNumber. This ensures no collision with regular relations
+ * even after OID wraparound.
+ */
+ locator->dbOid = (relform->relisshared ? InvalidOid : MyDatabaseId);
+ locator->spcOid = PSEUDO_PARTITION_TABLE_SPCOID;
+ locator->relNumber = relform->oid;
+ }
+
+ ReleaseSysCache(tuple);
+ return result;
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5f907335990..106d1e577a7 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -30,6 +30,7 @@
#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
#include "utils/wait_event.h"
@@ -1965,9 +1966,14 @@ Datum
pg_stat_reset_single_table_counters(PG_FUNCTION_ARGS)
{
Oid taboid = PG_GETARG_OID(0);
- Oid dboid = (IsSharedRelation(taboid) ? InvalidOid : MyDatabaseId);
+ RelFileLocator locator;
- pgstat_reset(PGSTAT_KIND_RELATION, dboid, taboid);
+ /* Get the stats locator from the relation OID */
+ if (!pgstat_reloid_to_relfilelocator(taboid, &locator))
+ PG_RETURN_VOID();
+
+ pgstat_reset(PGSTAT_KIND_RELATION, locator.dbOid,
+ RelFileLocatorToPgStatObjid(locator));
PG_RETURN_VOID();
}
@@ -2324,5 +2330,16 @@ pg_stat_have_stats(PG_FUNCTION_ARGS)
uint64 objid = PG_GETARG_INT64(2);
PgStat_Kind kind = pgstat_get_kind_from_str(stats_type);
+ /* Convert relation OID to relfilenode objid */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ RelFileLocator locator;
+
+ if (!pgstat_reloid_to_relfilelocator(objid, &locator))
+ PG_RETURN_BOOL(false);
+
+ objid = RelFileLocatorToPgStatObjid(locator);
+ }
+
PG_RETURN_BOOL(pgstat_have_entry(kind, dboid, objid));
}
diff --git a/src/include/catalog/pg_tablespace.dat b/src/include/catalog/pg_tablespace.dat
index c4cde415219..73ed046be31 100644
--- a/src/include/catalog/pg_tablespace.dat
+++ b/src/include/catalog/pg_tablespace.dat
@@ -10,6 +10,10 @@
#
#----------------------------------------------------------------------
+/*
+ * When adding a new one, ensure it does not conflict with
+ * PSEUDO_PARTITION_TABLE_SPCOID.
+ */
[
{ oid => '1663', oid_symbol => 'DEFAULTTABLESPACE_OID',
diff --git a/src/include/catalog/pg_tablespace.h b/src/include/catalog/pg_tablespace.h
index 3bd4a74f003..e5e818eb0c5 100644
--- a/src/include/catalog/pg_tablespace.h
+++ b/src/include/catalog/pg_tablespace.h
@@ -21,6 +21,14 @@
#include "catalog/genbki.h"
#include "catalog/pg_tablespace_d.h" /* IWYU pragma: export */
+/*
+ * Reserved tablespace OID for partitioned table pseudo locators.
+ * This is not an actual tablespace, just a reserved value to distinguish
+ * partitioned table statistics from regular table statistics. Ensures it does
+ * not conflict with the ones in pg_tablespace.dat.
+ */
+#define PSEUDO_PARTITION_TABLE_SPCOID 1665
+
/* ----------------
* pg_tablespace definition. cpp turns this into
* typedef struct FormData_pg_tablespace
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 216b93492ba..689c624f373 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -15,6 +15,7 @@
#include "portability/instr_time.h"
#include "postmaster/pgarch.h" /* for MAX_XFN_CHARS */
#include "replication/conflict.h"
+#include "storage/relfilelocator.h"
#include "utils/backend_progress.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
@@ -38,6 +39,12 @@ typedef struct RelationData *Relation;
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/*
+ * Build a pgstat key Objid based on a RelFileLocator.
+ */
+#define RelFileLocatorToPgStatObjid(locator) \
+ (((uint64) (locator).spcOid << 32) | (locator).relNumber)
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -178,11 +185,11 @@ typedef struct PgStat_TableCounts
*/
typedef struct PgStat_TableStatus
{
- Oid id; /* table's OID */
- bool shared; /* is it a shared catalog? */
+ uint64 id; /* hash of relfilelocator for stats key */
struct PgStat_TableXactStatus *trans; /* lowest subxact's counts */
PgStat_TableCounts counts; /* event counts to be sent */
Relation relation; /* rel that is using this entry */
+ RelFileLocator locator; /* table's relfilelocator */
} PgStat_TableStatus;
/* ----------
@@ -738,8 +745,8 @@ extern void pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
void *recdata, uint32 len);
extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry(Oid relid);
-extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry_ext(bool shared,
- Oid reloid);
+extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry_by_locator(RelFileLocator locator);
+extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry_ext(Oid reloid);
extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..42f29496551 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -765,6 +765,7 @@ extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
+extern bool pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator);
/*
diff --git a/src/test/recovery/t/029_stats_restart.pl b/src/test/recovery/t/029_stats_restart.pl
index cdc427dbc78..4d00087dc6f 100644
--- a/src/test/recovery/t/029_stats_restart.pl
+++ b/src/test/recovery/t/029_stats_restart.pl
@@ -55,10 +55,10 @@ trigger_funcrel_stat();
# verify stats objects exist
$sect = "initial";
-is(have_stats('database', $dboid, 0), 't', "$sect: db stats do exist");
-is(have_stats('function', $dboid, $funcoid),
+is(have_stats($connect_db, 'database', $dboid, 0), 't', "$sect: db stats do exist");
+is(have_stats($db_under_test, 'function', $dboid, $funcoid),
't', "$sect: function stats do exist");
-is(have_stats('relation', $dboid, $tableoid),
+is(have_stats($db_under_test, 'relation', $dboid, $tableoid),
't', "$sect: relation stats do exist");
# regular shutdown
@@ -79,10 +79,10 @@ copy($og_stats, $statsfile) or die "Copy failed: $!";
$node->start;
$sect = "copy";
-is(have_stats('database', $dboid, 0), 't', "$sect: db stats do exist");
-is(have_stats('function', $dboid, $funcoid),
+is(have_stats($connect_db, 'database', $dboid, 0), 't', "$sect: db stats do exist");
+is(have_stats($db_under_test, 'function', $dboid, $funcoid),
't', "$sect: function stats do exist");
-is(have_stats('relation', $dboid, $tableoid),
+is(have_stats($db_under_test, 'relation', $dboid, $tableoid),
't', "$sect: relation stats do exist");
$node->stop('immediate');
@@ -96,10 +96,10 @@ $node->start;
# stats should have been discarded
$sect = "post immediate";
-is(have_stats('database', $dboid, 0), 'f', "$sect: db stats do not exist");
-is(have_stats('function', $dboid, $funcoid),
+is(have_stats($connect_db, 'database', $dboid, 0), 'f', "$sect: db stats do not exist");
+is(have_stats($db_under_test, 'function', $dboid, $funcoid),
'f', "$sect: function stats do exist");
-is(have_stats('relation', $dboid, $tableoid),
+is(have_stats($db_under_test, 'relation', $dboid, $tableoid),
'f', "$sect: relation stats do not exist");
# get rid of backup statsfile
@@ -110,10 +110,10 @@ unlink $statsfile or die "cannot unlink $statsfile $!";
trigger_funcrel_stat();
$sect = "post immediate, new";
-is(have_stats('database', $dboid, 0), 't', "$sect: db stats do exist");
-is(have_stats('function', $dboid, $funcoid),
+is(have_stats($connect_db, 'database', $dboid, 0), 't', "$sect: db stats do exist");
+is(have_stats($db_under_test, 'function', $dboid, $funcoid),
't', "$sect: function stats do exist");
-is(have_stats('relation', $dboid, $tableoid),
+is(have_stats($db_under_test, 'relation', $dboid, $tableoid),
't', "$sect: relation stats do exist");
# regular shutdown
@@ -129,10 +129,10 @@ $node->start;
# no stats present due to invalid stats file
$sect = "invalid_overwrite";
-is(have_stats('database', $dboid, 0), 'f', "$sect: db stats do not exist");
-is(have_stats('function', $dboid, $funcoid),
+is(have_stats($connect_db, 'database', $dboid, 0), 'f', "$sect: db stats do not exist");
+is(have_stats($db_under_test, 'function', $dboid, $funcoid),
'f', "$sect: function stats do not exist");
-is(have_stats('relation', $dboid, $tableoid),
+is(have_stats($db_under_test, 'relation', $dboid, $tableoid),
'f', "$sect: relation stats do not exist");
@@ -145,10 +145,10 @@ append_file($og_stats, "XYZ");
$node->start;
$sect = "invalid_append";
-is(have_stats('database', $dboid, 0), 'f', "$sect: db stats do not exist");
-is(have_stats('function', $dboid, $funcoid),
+is(have_stats($connect_db, 'database', $dboid, 0), 'f', "$sect: db stats do not exist");
+is(have_stats($db_under_test, 'function', $dboid, $funcoid),
'f', "$sect: function stats do not exist");
-is(have_stats('relation', $dboid, $tableoid),
+is(have_stats($db_under_test, 'relation', $dboid, $tableoid),
'f', "$sect: relation stats do not exist");
@@ -307,9 +307,9 @@ sub trigger_funcrel_stat
sub have_stats
{
- my ($kind, $dboid, $objid) = @_;
+ my ($db, $kind, $dboid, $objid) = @_;
- return $node->safe_psql($connect_db,
+ return $node->safe_psql($db,
"SELECT pg_stat_have_stats('$kind', $dboid, $objid)");
}
--
2.34.1
--USlF2Vr25us5UpvY
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0002-handle-relation-statistics-correctly-during-rewr.patch"
^ permalink raw reply [nested|flat] 26+ messages in thread
* [PATCH v10 8/8] WIP: bufmgr: Rename ResOwnerReleaseBufferPin
@ 2026-01-13 00:28 Andres Freund <[email protected]>
0 siblings, 0 replies; 26+ messages in thread
From: Andres Freund @ 2026-01-13 00:28 UTC (permalink / raw)
This is separate as I'm not yet convinced of the new naming. The comment
probably makes sense regardless.
This is a name suggested a while ago by Melanie.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/buf_internals.h | 6 +++---
src/backend/storage/buffer/bufmgr.c | 22 ++++++++++++++--------
2 files changed, 17 insertions(+), 11 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 12086cf6dc7..b6714318154 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -520,18 +520,18 @@ extern PGDLLIMPORT CkptSortItem *CkptBufferIds;
/* ResourceOwner callbacks to hold buffer I/Os and pins */
extern PGDLLIMPORT const ResourceOwnerDesc buffer_io_resowner_desc;
-extern PGDLLIMPORT const ResourceOwnerDesc buffer_pin_resowner_desc;
+extern PGDLLIMPORT const ResourceOwnerDesc buffer_resowner_desc;
/* Convenience wrappers over ResourceOwnerRemember/Forget */
static inline void
ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer)
{
- ResourceOwnerRemember(owner, Int32GetDatum(buffer), &buffer_pin_resowner_desc);
+ ResourceOwnerRemember(owner, Int32GetDatum(buffer), &buffer_resowner_desc);
}
static inline void
ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer)
{
- ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_pin_resowner_desc);
+ ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_resowner_desc);
}
static inline void
ResourceOwnerRememberBufferIO(ResourceOwner owner, Buffer buffer)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e114e64fdd9..2f39454fd7f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -263,8 +263,8 @@ static void ForgetPrivateRefCountEntry(PrivateRefCountEntry *ref);
/* ResourceOwner callbacks to hold in-progress I/Os and buffer pins */
static void ResOwnerReleaseBufferIO(Datum res);
static char *ResOwnerPrintBufferIO(Datum res);
-static void ResOwnerReleaseBufferPin(Datum res);
-static char *ResOwnerPrintBufferPin(Datum res);
+static void ResOwnerReleaseBuffer(Datum res);
+static char *ResOwnerPrintBuffer(Datum res);
const ResourceOwnerDesc buffer_io_resowner_desc =
{
@@ -275,13 +275,13 @@ const ResourceOwnerDesc buffer_io_resowner_desc =
.DebugPrint = ResOwnerPrintBufferIO
};
-const ResourceOwnerDesc buffer_pin_resowner_desc =
+const ResourceOwnerDesc buffer_resowner_desc =
{
- .name = "buffer pin",
+ .name = "buffer",
.release_phase = RESOURCE_RELEASE_BEFORE_LOCKS,
.release_priority = RELEASE_PRIO_BUFFER_PINS,
- .ReleaseResource = ResOwnerReleaseBufferPin,
- .DebugPrint = ResOwnerPrintBufferPin
+ .ReleaseResource = ResOwnerReleaseBuffer,
+ .DebugPrint = ResOwnerPrintBuffer
};
/*
@@ -7671,8 +7671,14 @@ ResOwnerPrintBufferIO(Datum res)
return psprintf("lost track of buffer IO on buffer %d", buffer);
}
+/*
+ * Release buffer as part of resource owner cleanup. This will only be called
+ * if the buffer is pinned. If this backend held the content lock at the time
+ * of the error we also need to release that (note that it is not possible to
+ * hold a content lock without a pin).
+ */
static void
-ResOwnerReleaseBufferPin(Datum res)
+ResOwnerReleaseBuffer(Datum res)
{
Buffer buffer = DatumGetInt32(res);
@@ -7708,7 +7714,7 @@ ResOwnerReleaseBufferPin(Datum res)
}
static char *
-ResOwnerPrintBufferPin(Datum res)
+ResOwnerPrintBuffer(Datum res)
{
return DebugPrintBufferRefcount(DatumGetInt32(res));
}
--
2.48.1.76.g4e746b1a31.dirty
--jmawlk4t5yqwiemy--
^ permalink raw reply [nested|flat] 26+ messages in thread
end of thread, other threads:[~2026-01-13 00:28 UTC | newest]
Thread overview: 26+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-01-29 10:14 [PATCH v21 5/7] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH 09/12] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2021-02-10 10:57 [PATCH v24 08/10] Create custom compression methods Dilip Kumar <[email protected]>
2025-10-01 09:45 [PATCH v13 1/2] Key PGSTAT_KIND_RELATION by relfile locator Bertrand Drouvot <[email protected]>
2026-01-13 00:28 [PATCH v10 8/8] WIP: bufmgr: Rename ResOwnerReleaseBufferPin Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox