public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 01/12] Disallow compressed data inside container types
50+ messages / 2 participants
[nested] [flat]
* [PATCH 01/12] Disallow compressed data inside container types
@ 2021-02-16 13:54 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-02-16 13:54 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the comprassed field but while contructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we onlle decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
---
src/backend/access/common/heaptuple.c | 28 ++++++---------------------
src/backend/access/heap/heaptoast.c | 16 ++++++++-------
src/backend/executor/execTuples.c | 11 -----------
src/include/access/heaptoast.h | 4 ++--
4 files changed, 17 insertions(+), 42 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index 24a27e387d..5c3194f96d 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -983,30 +983,14 @@ heap_expand_tuple(HeapTuple sourceTuple, TupleDesc tupleDesc)
Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
- HeapTupleHeader td;
-
- /*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
- */
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
- tuple->t_len,
- tupleDesc);
-
/*
- * Fast path for easy case: just make a palloc'd copy and insert the
- * correct composite-Datum header fields (since those may not be set if
- * the given tuple came from disk, rather than from heap_form_tuple).
+ * The tuple contains compressed/external TOAST pointers, so we have
+ * to inline those fields to meet the conventions for composite-type
+ * Datums.
*/
- td = (HeapTupleHeader) palloc(tuple->t_len);
- memcpy((char *) td, (char *) tuple->t_data, tuple->t_len);
-
- HeapTupleHeaderSetDatumLength(td, tuple->t_len);
- HeapTupleHeaderSetTypeId(td, tupleDesc->tdtypeid);
- HeapTupleHeaderSetTypMod(td, tupleDesc->tdtypmod);
-
- return PointerGetDatum(td);
+ return toast_flatten_tuple_to_datum(tuple->t_data,
+ tuple->t_len,
+ tupleDesc);
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..dd162daab9 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -549,14 +549,15 @@ toast_flatten_tuple_to_datum(HeapTupleHeader tup,
/* ----------
* toast_build_flattened_tuple -
*
- * Build a tuple containing no out-of-line toasted fields.
- * (This does not eliminate compressed or short-header datums.)
+ * Build a tuple containing no compressed/out-of-line toasted fields.
+ * (This does not eliminate short-header datums.)
*
* This is essentially just like heap_form_tuple, except that it will
- * expand any external-data pointers beforehand.
+ * expand any compressed/external-data pointers beforehand.
*
- * It's not very clear whether it would be preferable to decompress
- * in-line compressed datums while at it. For now, we don't.
+ * It is not necessary to decompress the compressed data for the
+ * correctness, but reflects an expectation that compression will be more
+ * effective if applied to the whole tuple not individual fields.
* ----------
*/
HeapTuple
@@ -589,9 +590,10 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) ||
+ VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..ca7fbed576 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2189,13 +2189,6 @@ BuildTupleFromCStrings(AttInMetadata *attinmeta, char **values)
* memory context. Beware of code that changes context between the initial
* heap_form_tuple/etc call and calling HeapTuple(Header)GetDatum.
*
- * For performance-critical callers, it could be worthwhile to take extra
- * steps to ensure that there aren't TOAST pointers in the output of
- * heap_form_tuple to begin with. It's likely however that the costs of the
- * typcache lookup and tuple disassembly/reassembly are swamped by TOAST
- * dereference costs, so that the benefits of such extra effort would be
- * minimal.
- *
* XXX it would likely be better to create wrapper functions that produce
* a composite Datum from the field values in one step. However, there's
* enough code using the existing APIs that we couldn't get rid of this
@@ -2207,10 +2200,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 8b29f1a986..c80af45012 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -128,8 +128,8 @@ extern Datum toast_flatten_tuple_to_datum(HeapTupleHeader tup,
/* ----------
* toast_build_flattened_tuple -
*
- * Build a tuple containing no out-of-line toasted fields.
- * (This does not eliminate compressed or short-header datums.)
+ * Build a tuple containing no compressed/out-of-line toasted fields.
+ * (This does not eliminate short-header datums.)
* ----------
*/
extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
--
2.17.0
--9Ek0hoCL9XbhcSqy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0002-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Should consider materializing the cheapest inner path in consider_parallel_nestloop()
@ 2023-09-05 10:10 tender wang <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: tender wang @ 2023-09-05 10:10 UTC (permalink / raw)
To: [email protected]
After using patch, the result as below :
QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1078.00..26630101.20 rows=1 width=27) (actual
time=160571.005..160571.105 rows=0 loops=1)
InitPlan 1 (returns $0)
-> Limit (cost=78.00..78.00 rows=1 width=8) (actual
time=1.065..1.066 rows=1 loops=1)
-> Sort (cost=78.00..83.00 rows=2000 width=8) (actual
time=1.064..1.065 rows=1 loops=1)
Sort Key: part.p_partkey
Sort Method: top-N heapsort Memory: 25kB
-> Seq Scan on part (cost=0.00..68.00 rows=2000 width=8)
(actual time=0.046..0.830 rows=2000 loops=1)
-> Gather (cost=1000.00..26630023.20 rows=1 width=27) (actual
time=160571.003..160571.102 rows=0 loops=1)
Workers Planned: 1
Params Evaluated: $0
Workers Launched: 1
-> Nested Loop Left Join (cost=0.00..26629023.10 rows=1
width=27) (actual time=160549.257..160549.258 rows=0 loops=2)
Join Filter: ($0 IS NOT NULL)
Filter: ((sample_0.l_orderkey IS NULL) AND
(sample_0.l_shipmode IS NULL))
Rows Removed by Filter: 1810515312
-> Parallel Seq Scan on lineitem ref_0 (cost=0.00..1721.97
rows=35397 width=11) (actual time=0.010..3.393 rows=30088 loops=2)
-> Materialize (cost=0.00..2270.62 rows=60175 width=27)
(actual time=0.000..2.839 rows=60175 loops=60175)
-> Seq Scan on lineitem sample_0 (cost=0.00..1969.75
rows=60175 width=27) (actual time=0.008..11.381 rows=60175 loops=2)
Planning Time: 0.174 ms
Execution Time: 160571.476 ms
(20 rows)
tender wang <[email protected]> 于2023年9月5日周二 16:52写道：
> Hi all,
>
> I recently run benchmark[1] on master, but I found performance problem
> as below:
>
> explain analyze select
> subq_0.c0 as c0,
> subq_0.c1 as c1,
> subq_0.c2 as c2
> from
> (select
> ref_0.l_shipmode as c0,
> sample_0.l_orderkey as c1,
> sample_0.l_quantity as c2,
> ref_0.l_orderkey as c3,
> sample_0.l_shipmode as c5,
> ref_0.l_shipinstruct as c6
> from
> public.lineitem as ref_0
> left join public.lineitem as sample_0
> on ((select p_partkey from public.part order by p_partkey limit
> 1)
> is not NULL)
> where sample_0.l_orderkey is NULL) as subq_0
> where subq_0.c5 is NULL
> limit 1;
> QUERY PLAN
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------
> Limit (cost=78.00..45267050.75 rows=1 width=27) (actual
> time=299695.097..299695.099 rows=0 loops=1)
> InitPlan 1 (returns $0)
> -> Limit (cost=78.00..78.00 rows=1 width=8) (actual
> time=0.651..0.652 rows=1 loops=1)
> -> Sort (cost=78.00..83.00 rows=2000 width=8) (actual
> time=0.650..0.651 rows=1 loops=1)
> Sort Key: part.p_partkey
> Sort Method: top-N heapsort Memory: 25kB
> -> Seq Scan on part (cost=0.00..68.00 rows=2000
> width=8) (actual time=0.013..0.428 rows=2000 loops=1)
> -> Nested Loop Left Join (cost=0.00..45266972.75 rows=1 width=27)
> (actual time=299695.096..299695.096 rows=0 loops=1)
> Join Filter: ($0 IS NOT NULL)
> Filter: ((sample_0.l_orderkey IS NULL) AND (sample_0.l_shipmode
> IS NULL))
> Rows Removed by Filter: 3621030625
> -> Seq Scan on lineitem ref_0 (cost=0.00..1969.75 rows=60175
> width=11) (actual time=0.026..6.225 rows=60175 loops=1)
> -> Materialize (cost=0.00..2270.62 rows=60175 width=27) (actual
> time=0.000..2.554 rows=60175 loops=60175)
> -> Seq Scan on lineitem sample_0 (cost=0.00..1969.75
> rows=60175 width=27) (actual time=0.004..8.169 rows=60175 loops=1)
> Planning Time: 0.172 ms
> Execution Time: 299695.501 ms
> (16 rows)
>
> After I set enable_material to off, the same query run faster, as below:
> set enable_material = off;
> explain analyze select
> subq_0.c0 as c0,
> subq_0.c1 as c1,
> subq_0.c2 as c2
> from
> (select
> ref_0.l_shipmode as c0,
> sample_0.l_orderkey as c1,
> sample_0.l_quantity as c2,
> ref_0.l_orderkey as c3,
> sample_0.l_shipmode as c5,
> ref_0.l_shipinstruct as c6
> from
> public.lineitem as ref_0
> left join public.lineitem as sample_0
> on ((select p_partkey from public.part order by p_partkey limit
> 1)
> is not NULL)
> where sample_0.l_orderkey is NULL) as subq_0
> where subq_0.c5 is NULL
> limit 1;
> QUERY
> PLAN
>
> -----------------------------------------------------------------------------------------------------------------------------------------------
> Limit (cost=1078.00..91026185.57 rows=1 width=27) (actual
> time=192669.605..192670.425 rows=0 loops=1)
> InitPlan 1 (returns $0)
> -> Limit (cost=78.00..78.00 rows=1 width=8) (actual
> time=0.662..0.663 rows=1 loops=1)
> -> Sort (cost=78.00..83.00 rows=2000 width=8) (actual
> time=0.661..0.662 rows=1 loops=1)
> Sort Key: part.p_partkey
> Sort Method: top-N heapsort Memory: 25kB
> -> Seq Scan on part (cost=0.00..68.00 rows=2000
> width=8) (actual time=0.017..0.430 rows=2000 loops=1)
> -> Gather (cost=1000.00..91026107.57 rows=1 width=27) (actual
> time=192669.604..192670.422 rows=0 loops=1)
> Workers Planned: 1
> Params Evaluated: $0
> Workers Launched: 1
> -> Nested Loop Left Join (cost=0.00..91025107.47 rows=1
> width=27) (actual time=192588.143..192588.144 rows=0 loops=2)
> Join Filter: ($0 IS NOT NULL)
> Filter: ((sample_0.l_orderkey IS NULL) AND
> (sample_0.l_shipmode IS NULL))
> Rows Removed by Filter: 1810515312
> -> Parallel Seq Scan on lineitem ref_0
> (cost=0.00..1721.97 rows=35397 width=11) (actual time=0.007..3.797
> rows=30088 loops=2)
> -> Seq Scan on lineitem sample_0 (cost=0.00..1969.75
> rows=60175 width=27) (actual time=0.000..2.637 rows=60175 loops=60175)
> Planning Time: 0.174 ms
> Execution Time: 192670.458 ms
> (19 rows)
>
> I debug the code and find consider_parallel_nestloop() doesn't consider
> materialized form of the cheapest inner path.
> When enable_material = true, we can see Material path won in first plan,
> but Parallel Seq Scan node doesn't add as outer path, which because
> in try_partial_nestloop_path() , the cost of nestloop wat computed using
> seq scan path not material path.
>
> [1] include test table schema and data, you can repeat above problem.
>
> I try fix this problem in attached patch, and I found pg12.12 also had
> this issue. Please review my patch, thanks!
>
> [1] https://github.com/tenderwg/tpch_test
>
^ permalink raw reply [nested|flat] 50+ messages in thread
end of thread, other threads:[~2023-09-05 10:10 UTC | newest]
Thread overview: 50+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-16 13:54 [PATCH 01/12] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2023-09-05 10:10 Re: Should consider materializing the cheapest inner path in consider_parallel_nestloop() tender wang <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox