From: Michael Paquier <michael@paquier.xyz>
To: Burd, Greg <greg@burd.me>
Cc: Nikita Malakhov <hukutoc@gmail.com>
Cc: Hannu Krosing <hannuk@google.com>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)
Date: Fri, 8 Aug 2025 16:51:45 +0900
Message-ID: <aJWska1_ZiXUIETX@paquier.xyz> (raw)
In-Reply-To: <aIyCz4T858Kcm4UU@paquier.xyz>
References: <aFOnKHG7Wn-Srnpv@paquier.xyz>
<CAMT0RQQvki5v5FPZMXSR3cn4Y2gMUUtEYb+Hf_OQmqUbeBrUeQ@mail.gmail.com>
<CAN-LCVNsE4x0k11ZRWvU4ySTbe98fwA16qzV7p8dxogWnD5Jng@mail.gmail.com>
<aGh7xCegTRJCdQ9q@paquier.xyz>
<CAN-LCVNioWbZgPvw=grtjQDgZG1H3Dgv8udb=7N-9JvhsXfwnw@mail.gmail.com>
<aGxagUqel7d6A6Xm@paquier.xyz>
<783176F8-1F6C-444C-8997-6DD05A4A3B10@burd.me>
<aG2nv0Iw9Te7c6en@paquier.xyz>
<aIyCz4T858Kcm4UU@paquier.xyz>
On Fri, Aug 01, 2025 at 06:03:11PM +0900, Michael Paquier wrote:
> Please find attached a v3, that I have spent some time polishing to
> fix the value ID problem of this thread. v2 had some conflicts, and
> the CI previously failed with warning job (CI is green here now).
Attached is a v4, due to conflicts mainly caused by the recent changes
in varatt.h done by e035863c9a04. This had an interesting side
benefit when rebasing, where I have been able to isolate most of the
knowledge related to the struct varatt_external (well
varatt_external_oid in the patch set) into toast_external.c, at the
exception of VARTAG_SIZE. That's done in a separate patch, numbered
0006.
The rest of the patch set has a couple of adjustements to document
better the new API expectations for toast_external.{c,h}, comment
adjustments, some more beautification changes, some indentation
applied, etc.
> As things stand, I am getting pretty happy with the patch set up to
> 0005 and how things are getting in shape for the interface, and I am
> planning to begin applying this stuff up to 0005 in the next couple of
> weeks.
As of this patch set, this means a new target of 0006, to get the
TOAST code refactored so as it is able to support more than 1 type of
external on-disk pointer with the 8-byte value problem in scope. Any
comments?
--
Michael
Attachments:download all (16 files, .tar.gz)v4-0001-Refactor-some-TOAST-value-ID-code-to-use-uint64-i.patch (text/x-diff, 16.0K) download
From f10c5b42c0c990897a8a654828d90f0304330b3e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 19 Jun 2025 09:57:19 +0900
Subject: [PATCH v4 01/15] Refactor some TOAST value ID code to use uint64
instead of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become uint64, easing an upcoming
change to allow 8-byte TOAST values.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to PRIu64.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 14 ++--
contrib/amcheck/verify_heapam.c | 69 +++++++++++--------
6 files changed, 62 insertions(+), 47 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.hindex 6385a27caf83..6e3558cbd6d2 100644--- a/src/include/access/heaptoast.h+++ b/src/include/access/heaptoast.h@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,+extern void heap_fetch_toast_slice(Relation toastrel, uint64 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, struct varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.hindex 1c9e802a6b12..b640047d2fdc 100644--- a/src/include/access/tableam.h+++ b/src/include/access/tableam.h@@ -740,7 +740,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,+ void (*relation_fetch_toast_slice) (Relation toastrel, uint64 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1873,7 +1873,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,+table_relation_fetch_toast_slice(Relation toastrel, uint64 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, struct varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex 196e06115e93..0016a16afc95 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);+static bool toastrel_valueid_exists(Relation toastrel, uint64 valueid);+static bool toastid_valueid_exists(Oid toastrelid, uint64 valueid);
/* ----------
* toast_compress_datum -
@@ -456,7 +456,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)+toastrel_valueid_exists(Relation toastrel, uint64 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -504,7 +504,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)+toastid_valueid_exists(Oid toastrelid, uint64 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex cb1e57030f64..76936b2f4944 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,+heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
struct varlena *result)
{
@@ -725,7 +725,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",+ elog(ERROR, "found toasted toast chunk for toast value %" PRIu64 " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -737,13 +737,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value %" PRIu64 " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %" PRIu64 " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -752,7 +752,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %" PRIu64 " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -781,7 +781,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",+ errmsg_internal("missing chunk number %d for toast value %" PRIu64 " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.cindex 34cf05668ae8..36e2a4360e18 100644--- a/src/backend/replication/logical/reorderbuffer.c+++ b/src/backend/replication/logical/reorderbuffer.c@@ -176,7 +176,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */+ uint64 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -4944,7 +4944,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);+ hash_ctl.keysize = sizeof(uint64);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -4968,7 +4968,7 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;+ uint64 chunk_id;
int32 chunk_seq;
if (txn->toast_hash == NULL)
@@ -4995,11 +4995,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",+ elog(ERROR, "got sequence entry %d for toast chunk %" PRIu64 " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",+ elog(ERROR, "got sequence entry %d for toast chunk %" PRIu64 " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5108,6 +5108,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
struct varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ uint64 toast_valueid;
/* system columns aren't toasted */
if (attr->attnum < 0)
@@ -5132,13 +5133,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 4963e9245cb5..3b2bdced4cdc 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -1556,11 +1556,18 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 extsize)
{
int32 chunk_seq;
- int32 last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;+ int32 last_chunk_seq;
Pointer chunk;
bool isnull;
int32 chunksize;
int32 expected_size;
+ uint64 toast_valueid;+ int32 max_chunk_size;++ toast_valueid = ta->toast_pointer.va_valueid;++ max_chunk_size = TOAST_MAX_CHUNK_SIZE;+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1568,16 +1575,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",- ta->toast_pointer.va_valueid));+ psprintf("toast value %" PRIu64 " has toast chunk with null sequence number",+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",- ta->toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " index scan returned chunk %d when expecting chunk %d",+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1588,8 +1595,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",- ta->toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " chunk %d has null data",+ toast_valueid,
chunk_seq));
return;
}
@@ -1608,8 +1615,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",- ta->toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " chunk %d has invalid varlena header %0x",+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1620,19 +1627,19 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",- ta->toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " chunk %d follows last expected chunk %d",+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
- expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE- : extsize - (last_chunk_seq * TOAST_MAX_CHUNK_SIZE);+ expected_size = chunk_seq < last_chunk_seq ? max_chunk_size+ : extsize - (last_chunk_seq * max_chunk_size);
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",- ta->toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " chunk %d has size %u, but expected size %u",+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1663,6 +1670,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
struct varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ uint64 toast_pointer_valueid;
CompactAttribute *thisatt;
struct varatt_external toast_pointer;
@@ -1766,6 +1774,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
return true;
/* It is external, and we're looking at a page on disk */
+ toast_pointer_valueid = toast_pointer.va_valueid;
/*
* Must copy attr into toast_pointer for alignment considerations
@@ -1775,8 +1784,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",- toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " rawsize %d exceeds limit %d",+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1803,16 +1812,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",- toast_pointer.va_valueid, cmid));+ psprintf("toast value %" PRIu64 " has invalid compression method id %d",+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",- toast_pointer.va_valueid));+ psprintf("toast value %" PRIu64 " is external but tuple header flag HEAP_HASEXTERNAL not set",+ toast_pointer_valueid));
return true;
}
@@ -1820,8 +1829,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",- toast_pointer.va_valueid));+ psprintf("toast value %" PRIu64 " is external but relation has no toast relation",+ toast_pointer_valueid));
return true;
}
@@ -1866,9 +1875,11 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
+ uint64 toast_valueid;+ int32 max_chunk_size = TOAST_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
@@ -1896,14 +1907,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",- ta->toast_pointer.va_valueid));+ psprintf("toast value %" PRIu64 " not found in toast table",+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",- ta->toast_pointer.va_valueid,+ psprintf("toast value %" PRIu64 " was expected to end at chunk %d, but ended while expecting chunk %d",+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.50.0
v4-0002-Minimize-footprint-of-TOAST_MAX_CHUNK_SIZE-in-hea.patch (text/x-diff, 2.9K) download
From a25391ebd3bc27dcffd7119b86e7632ace01b512 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 19 Jun 2025 10:03:46 +0900
Subject: [PATCH v4 02/15] Minimize footprint of TOAST_MAX_CHUNK_SIZE in heap
TOAST code
This eases a follow-up change to support 8-byte TOAST value IDs, as the
maximum chunk size allowed for a single chunk of TOASTed data depends on
the size of the value ID.
---
src/backend/access/heap/heaptoast.c | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex 76936b2f4944..ae8d502ddcd3 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -634,11 +634,12 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
SysScanDesc toastscan;
HeapTuple ttup;
int32 expectedchunk;
- int32 totalchunks = ((attrsize - 1) / TOAST_MAX_CHUNK_SIZE) + 1;+ int32 totalchunks;
int startchunk;
int endchunk;
int num_indexes;
int validIndex;
+ int32 max_chunk_size;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -646,8 +647,11 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
- startchunk = sliceoffset / TOAST_MAX_CHUNK_SIZE;- endchunk = (sliceoffset + slicelength - 1) / TOAST_MAX_CHUNK_SIZE;+ max_chunk_size = TOAST_MAX_CHUNK_SIZE;++ totalchunks = ((attrsize - 1) / max_chunk_size) + 1;+ startchunk = sliceoffset / max_chunk_size;+ endchunk = (sliceoffset + slicelength - 1) / max_chunk_size;
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
@@ -747,8 +751,8 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
- expected_size = curchunk < totalchunks - 1 ? TOAST_MAX_CHUNK_SIZE- : attrsize - ((totalchunks - 1) * TOAST_MAX_CHUNK_SIZE);+ expected_size = curchunk < totalchunks - 1 ? max_chunk_size+ : attrsize - ((totalchunks - 1) * max_chunk_size);
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -763,12 +767,12 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
chcpystrt = 0;
chcpyend = chunksize - 1;
if (curchunk == startchunk)
- chcpystrt = sliceoffset % TOAST_MAX_CHUNK_SIZE;+ chcpystrt = sliceoffset % max_chunk_size;
if (curchunk == endchunk)
- chcpyend = (sliceoffset + slicelength - 1) % TOAST_MAX_CHUNK_SIZE;+ chcpyend = (sliceoffset + slicelength - 1) % max_chunk_size;
memcpy(VARDATA(result) +
- (curchunk * TOAST_MAX_CHUNK_SIZE - sliceoffset) + chcpystrt,+ (curchunk * max_chunk_size - sliceoffset) + chcpystrt,
chunkdata + chcpystrt,
(chcpyend - chcpystrt) + 1);
--
2.50.0
v4-0003-varatt_external-varatt_external_oid-and-VARTAG_ON.patch (text/x-diff, 12.5K) download
From 742631089dfe7f2f247ee2cd652e749081d68c47 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 14:50:06 +0900
Subject: [PATCH v4 03/15] varatt_external->varatt_external_oid and
VARTAG_ONDISK->VARTAG_ONDISK_OID
This rename is in preparation of a follow-up commit that aims at adding
support for multiple types of external on-disk TOAST pointers, where the
OID type is only one subset of them.
---
src/include/access/detoast.h | 4 +--
src/include/varatt.h | 34 +++++++++++--------
src/backend/access/common/detoast.c | 10 +++---
src/backend/access/common/toast_compression.c | 2 +-
src/backend/access/common/toast_internals.c | 8 ++---
.../replication/logical/reorderbuffer.c | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
contrib/amcheck/verify_heapam.c | 6 ++--
8 files changed, 36 insertions(+), 32 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.hindex e603a2276c38..d80a62e64fd5 100644--- a/src/include/access/detoast.h+++ b/src/include/access/detoast.h@@ -14,7 +14,7 @@
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
- * into a local "struct varatt_external" toast pointer. This should be+ * into a local "struct varatt_external_oid" toast pointer. This should be
* just a memcpy, but some versions of gcc seem to produce broken code
* that assumes the datum contents are aligned. Introducing an explicit
* intermediate "varattrib_1b_e *" variable seems to fix it.
@@ -28,7 +28,7 @@ do { \
} while (0)
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
-#define TOAST_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external))+#define TOAST_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/varatt.h b/src/include/varatt.hindex aeeabf9145b5..fcf2dffaa2fd 100644--- a/src/include/varatt.h+++ b/src/include/varatt.h@@ -16,7 +16,7 @@
#define VARATT_H
/*
- * struct varatt_external is a traditional "TOAST pointer", that is, the+ * struct varatt_external_oid is a traditional "TOAST pointer", that is, the
* information needed to fetch a Datum stored out-of-line in a TOAST table.
* The data is compressed if and only if the external size stored in
* va_extinfo is less than va_rawsize - VARHDRSZ.
@@ -29,14 +29,14 @@
* you can look at these fields! (The reason we use memcmp is to avoid
* having to do that just to detect equality of two TOAST pointers...)
*/
-typedef struct varatt_external+typedef struct varatt_external_oid
{
int32 va_rawsize; /* Original data size (includes header) */
uint32 va_extinfo; /* External saved size (without header) and
* compression method */
Oid va_valueid; /* Unique ID of value within TOAST table */
Oid va_toastrelid; /* RelID of TOAST table containing it */
-} varatt_external;+} varatt_external_oid;
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
@@ -51,7 +51,7 @@ typedef struct varatt_external
* The creator of such a Datum is entirely responsible that the referenced
* storage survives for as long as referencing pointer Datums can exist.
*
- * Note that just as for struct varatt_external, this struct is stored+ * Note that just as for struct varatt_external_oid, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct varatt_indirect
@@ -66,7 +66,7 @@ typedef struct varatt_indirect
* storage. APIs for this, in particular the definition of struct
* ExpandedObjectHeader, are in src/include/utils/expandeddatum.h.
*
- * Note that just as for struct varatt_external, this struct is stored+ * Note that just as for struct varatt_external_oid, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct ExpandedObjectHeader ExpandedObjectHeader;
@@ -78,15 +78,16 @@ typedef struct varatt_expanded
/*
* Type tag for the various sorts of "TOAST pointer" datums. The peculiar
- * value for VARTAG_ONDISK comes from a requirement for on-disk compatibility- * with a previous notion that the tag field was the pointer datum's length.+ * value for VARTAG_ONDISK_OID comes from a requirement for on-disk+ * compatibility with a previous notion that the tag field was the pointer+ * datum's length.
*/
typedef enum vartag_external
{
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
- VARTAG_ONDISK = 18+ VARTAG_ONDISK_OID = 18
} vartag_external;
/* Is a TOAST pointer either type of expanded-object pointer? */
@@ -105,8 +106,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_indirect);
else if (VARTAG_IS_EXPANDED(tag))
return sizeof(varatt_expanded);
- else if (tag == VARTAG_ONDISK)- return sizeof(varatt_external);+ else if (tag == VARTAG_ONDISK_OID)+ return sizeof(varatt_external_oid);
else
{
Assert(false);
@@ -360,7 +361,7 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK;+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
/* Is varlena datum an indirect pointer? */
@@ -502,15 +503,18 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
return ((varattrib_4b *) PTR)->va_compressed.va_tcinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Same for external Datums; but note argument is a struct varatt_external */+/*+ * Same for external Datums; but note argument is a struct+ * varatt_external_oid.+ */
static inline Size
-VARATT_EXTERNAL_GET_EXTSIZE(struct varatt_external toast_pointer)+VARATT_EXTERNAL_GET_EXTSIZE(struct varatt_external_oid toast_pointer)
{
return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
}
static inline uint32
-VARATT_EXTERNAL_GET_COMPRESS_METHOD(struct varatt_external toast_pointer)+VARATT_EXTERNAL_GET_COMPRESS_METHOD(struct varatt_external_oid toast_pointer)
{
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
@@ -533,7 +537,7 @@ VARATT_EXTERNAL_GET_COMPRESS_METHOD(struct varatt_external toast_pointer)
* actually saves space, so we expect either equality or less-than.
*/
static inline bool
-VARATT_EXTERNAL_IS_COMPRESSED(struct varatt_external toast_pointer)+VARATT_EXTERNAL_IS_COMPRESSED(struct varatt_external_oid toast_pointer)
{
return VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) <
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.cindex 626517877422..04eedb474c74 100644--- a/src/backend/access/common/detoast.c+++ b/src/backend/access/common/detoast.c@@ -225,7 +225,7 @@ detoast_attr_slice(struct varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
@@ -344,7 +344,7 @@ toast_fetch_datum(struct varlena *attr)
{
Relation toastrel;
struct varlena *result;
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
int32 attrsize;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
@@ -398,7 +398,7 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
{
Relation toastrel;
struct varlena *result;
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
int32 attrsize;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
@@ -550,7 +550,7 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
result = toast_pointer.va_rawsize;
@@ -610,7 +610,7 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
result = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.cindex 926f1e4008ab..26aad84d367a 100644--- a/src/backend/access/common/toast_compression.c+++ b/src/backend/access/common/toast_compression.c@@ -262,7 +262,7 @@ toast_get_compression_id(struct varlena *attr)
*/
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex 0016a16afc95..d555c02e908e 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -127,7 +127,7 @@ toast_save_datum(Relation rel, Datum value,
bool t_isnull[3];
CommandId mycid = GetCurrentCommandId(true);
struct varlena *result;
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
union
{
struct varlena hdr;
@@ -237,7 +237,7 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_valueid = InvalidOid;
if (oldexternal != NULL)
{
- struct varatt_external old_toast_pointer;+ struct varatt_external_oid old_toast_pointer;
Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
/* Must copy to access aligned fields */
@@ -369,7 +369,7 @@ toast_save_datum(Relation rel, Datum value,
* Create the TOAST pointer value that we'll return
*/
result = (struct varlena *) palloc(TOAST_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK);+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
return PointerGetDatum(result);
@@ -385,7 +385,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
struct varlena *attr = (struct varlena *) DatumGetPointer(value);
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.cindex 36e2a4360e18..0eb99b4f1377 100644--- a/src/backend/replication/logical/reorderbuffer.c+++ b/src/backend/replication/logical/reorderbuffer.c@@ -5102,7 +5102,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
struct varlena *varlena;
/* va_rawsize is the size of the original datum -- including header */
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
struct varatt_indirect redirect_pointer;
struct varlena *new_datum = NULL;
struct varlena *reconstructed;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.cindex ffae8c23abfa..38d4e6d45f28 100644--- a/src/backend/utils/adt/varlena.c+++ b/src/backend/utils/adt/varlena.c@@ -4219,7 +4219,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
struct varlena *attr;
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 3b2bdced4cdc..030f9fb64b51 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -73,7 +73,7 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1672,7 +1672,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
uint16 infomask;
uint64 toast_pointer_valueid;
CompactAttribute *thisatt;
- struct varatt_external toast_pointer;+ struct varatt_external_oid toast_pointer;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1731,7 +1731,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK)+ if (va_tag != VARTAG_ONDISK_OID)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
--
2.50.0
v4-0004-Refactor-external-TOAST-pointer-code-for-better-p.patch (text/x-diff, 44.2K) download
From 39305f534c19648592b955b8bdc265ae16e50f67 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 15:31:19 +0900
Subject: [PATCH v4 04/15] Refactor external TOAST pointer code for better
pluggability
This commit introduces a new interface for external TOAST pointers,
which is able to make a translation of the varlena pointers stored on
disk to/from an new in-memory structure called toast_external. The
types of varatt_external supported on disk need to be registered into a
new subsystem in a new file, called toast_external.[c|h], then define a
set of callbacks to allow the toasting and detoasting code to use it.
A follow-up change will rely on this refactoring to introduce new
vartag_external values with an associated varatt_external_* that is
able, which would be used in int8 TOAST tables.
---
src/include/access/detoast.h | 12 +-
src/include/access/heaptoast.h | 5 +-
src/include/access/toast_external.h | 176 ++++++++++++++++++
src/include/access/toast_helper.h | 1 +
src/include/varatt.h | 16 +-
src/backend/access/common/Makefile | 1 +
src/backend/access/common/detoast.c | 57 +++---
src/backend/access/common/meson.build | 1 +
src/backend/access/common/toast_compression.c | 10 +-
src/backend/access/common/toast_external.c | 161 ++++++++++++++++
src/backend/access/common/toast_internals.c | 84 ++++++---
src/backend/access/heap/heaptoast.c | 20 +-
src/backend/access/table/toast_helper.c | 12 +-
.../replication/logical/reorderbuffer.c | 13 +-
src/backend/utils/adt/varlena.c | 7 +-
doc/src/sgml/storage.sgml | 2 +-
contrib/amcheck/verify_heapam.c | 35 ++--
src/tools/pgindent/typedefs.list | 2 +
18 files changed, 507 insertions(+), 108 deletions(-)
create mode 100644 src/include/access/toast_external.h
create mode 100644 src/backend/access/common/toast_external.c
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.hindex d80a62e64fd5..4195f7b5bdfd 100644--- a/src/include/access/detoast.h+++ b/src/include/access/detoast.h@@ -14,10 +14,11 @@
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
- * into a local "struct varatt_external_oid" toast pointer. This should be- * just a memcpy, but some versions of gcc seem to produce broken code- * that assumes the datum contents are aligned. Introducing an explicit- * intermediate "varattrib_1b_e *" variable seems to fix it.+ * into a local "struct varatt_external_*" toast pointer, as supported+ * in toast_external.c and varatt.h. This should be just a memcpy, but+ * some versions of gcc seem to produce broken code that assumes the datum+ * contents are aligned. Introducing an explicit intermediate+ * "varattrib_1b_e *" variable seems to fix it.
*/
#define VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr) \
do { \
@@ -27,9 +28,6 @@ do { \
memcpy(&(toast_pointer), VARDATA_EXTERNAL(attre), sizeof(toast_pointer)); \
} while (0)
-/* Size of an EXTERNAL datum that contains a standard TOAST pointer */-#define TOAST_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))-
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.hindex 6e3558cbd6d2..673e96f5488c 100644--- a/src/include/access/heaptoast.h+++ b/src/include/access/heaptoast.h@@ -81,13 +81,16 @@
#define EXTERN_TUPLE_MAX_SIZE MaximumBytesPerTuple(EXTERN_TUPLES_PER_PAGE)
-#define TOAST_MAX_CHUNK_SIZE \+#define TOAST_MAX_CHUNK_SIZE_OID \
(EXTERN_TUPLE_MAX_SIZE - \
MAXALIGN(SizeofHeapTupleHeader) - \
sizeof(Oid) - \
sizeof(int32) - \
VARHDRSZ)
+/* Maximum size of chunk possible */+#define TOAST_MAX_CHUNK_SIZE TOAST_MAX_CHUNK_SIZE_OID+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/access/toast_external.h b/src/include/access/toast_external.hnew file mode 100644index 000000000000..ad2a10f35c01--- /dev/null+++ b/src/include/access/toast_external.h@@ -0,0 +1,176 @@+/*-------------------------------------------------------------------------+ *+ * toast_external.h+ * Support for on-disk external TOAST pointers+ *+ *+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group+ * Portions Copyright (c) 1995, Regents of the University of California+ *+ * src/include/access/toast_external.h+ *+ *-------------------------------------------------------------------------+ */++#ifndef TOAST_EXTERNAL_H+#define TOAST_EXTERNAL_H++#include "access/toast_compression.h"+#include "varatt.h"++/*+ * Intermediate in-memory structure used when creating on-disk+ * varatt_external_* or when deserializing varlena contents.+ */+typedef struct toast_external_data+{+ /* Original data size (includes header) */+ int32 rawsize;++ /* External saved size (without header) */+ uint32 extsize;++ /*+ * Compression method.+ *+ * If not compressed, set to TOAST_INVALID_COMPRESSION_ID.+ */+ ToastCompressionId compression_method;++ /* Relation OID of TOAST table containing the value */+ Oid toastrelid;++ /*+ * Unique ID of value within TOAST table. This could be an OID or an int8+ * value. This field is large enough to be able to store any of these.+ */+ uint64 value;+} toast_external_data;++/*+ * Metadata for external TOAST pointer kinds, separated based on their+ * vartag_external.+ */+typedef struct toast_external_info+{+ /*+ * Maximum chunk of data authorized for this type of external TOAST+ * pointer, when dividing an entry by chunks. Sized depending on the size+ * of its varatt_external_* structure.+ */+ int32 maximum_chunk_size;++ /*+ * Size of an external TOAST pointer of this type, typically+ * (VARHDRSZ_EXTERNAL + sizeof(varatt_external_struct)).+ */+ int32 toast_pointer_size;++ /*+ * Map an input varlena to a toast_external_data, for consumption in the+ * backend code. "data" is an input/output result.+ */+ void (*to_external_data) (struct varlena *attr,+ toast_external_data *data);++ /*+ * Create a varlena that will be used on-disk for the given TOAST type,+ * based on the given input data.+ *+ * The result is the varlena created, for on-disk insertion.+ */+ struct varlena *(*create_external_data) (toast_external_data data);++} toast_external_info;++/* Retrieve a toast_external_info from a vartag */+extern const toast_external_info *toast_external_get_info(uint8 tag);++/* Retrieve toast_pointer_size using a TOAST attribute type */+extern int32 toast_external_info_get_pointer_size(uint8 tag);++/* Retrieve the vartag to assign to a TOAST typle */+extern uint8 toast_external_assign_vartag(Oid toastrelid, uint64 value);++/*+ * Testing whether an externally-stored value is compressed now requires+ * comparing size stored in extsize (the actual length of the external data)+ * to rawsize (the original uncompressed datum's size). The latter includes+ * VARHDRSZ overhead, the former doesn't. We never use compression unless it+ * actually saves space, so we expect either equality or less-than.+ */+static inline bool+TOAST_EXTERNAL_IS_COMPRESSED(toast_external_data data)+{+ return data.extsize < (data.rawsize - VARHDRSZ);+}++/* Full data structure */+static inline void+toast_external_info_get_data(struct varlena *attr, toast_external_data *data)+{+ uint8 tag = VARTAG_EXTERNAL(attr);+ const toast_external_info *info = toast_external_get_info(tag);++ info->to_external_data(attr, data);+}++/*+ * Helper routines to recover specific fields in toast_external_data. Most+ * code paths doing work with on-disk external TOAST pointers care about+ * these.+ */++/* Detoasted "raw" size */+static inline Size+toast_external_info_get_rawsize(struct varlena *attr)+{+ uint8 tag = VARTAG_EXTERNAL(attr);+ const toast_external_info *info = toast_external_get_info(tag);+ toast_external_data data;++ info->to_external_data(attr, &data);++ return data.rawsize;+}++/* External saved size */+static inline Size+toast_external_info_get_extsize(struct varlena *attr)+{+ uint8 tag = VARTAG_EXTERNAL(attr);+ const toast_external_info *info = toast_external_get_info(tag);+ toast_external_data data;++ info->to_external_data(attr, &data);++ return data.extsize;+}++/* Compression method ID */+static inline ToastCompressionId+toast_external_info_get_compression_method(struct varlena *attr)+{+ uint8 tag = VARTAG_EXTERNAL(attr);+ const toast_external_info *info = toast_external_get_info(tag);+ toast_external_data data;++ info->to_external_data(attr, &data);++ return data.compression_method;+}++/* Value ID */+static inline Size+toast_external_info_get_value(struct varlena *attr)+{+ uint8 tag = VARTAG_EXTERNAL(attr);+ const toast_external_info *info = toast_external_get_info(tag);+ toast_external_data data;++ info->to_external_data(attr, &data);++ return data.value;+}++#endif /* TOAST_EXTERNAL_H */diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.hindex e6ab8afffb67..6bc912809f34 100644--- a/src/include/access/toast_helper.h+++ b/src/include/access/toast_helper.h@@ -47,6 +47,7 @@ typedef struct
* should be NULL in the case of an insert.
*/
Relation ttc_rel; /* the relation that contains the tuple */
+ int32 ttc_toast_pointer_size; /* size of external TOAST pointer */
Datum *ttc_values; /* values from the tuple columns */
bool *ttc_isnull; /* null flags for the tuple columns */
Datum *ttc_oldvalues; /* values from previous tuple */
diff --git a/src/include/varatt.h b/src/include/varatt.hindex fcf2dffaa2fd..b791ce7847ed 100644--- a/src/include/varatt.h+++ b/src/include/varatt.h@@ -21,6 +21,9 @@
* The data is compressed if and only if the external size stored in
* va_extinfo is less than va_rawsize - VARHDRSZ.
*
+ * The value ID is an OID, used for TOAST relations with OID as attribute+ * for chunk_id.+ *
* This struct must not contain any padding, because we sometimes compare
* these pointers using memcmp.
*
@@ -51,7 +54,7 @@ typedef struct varatt_external_oid
* The creator of such a Datum is entirely responsible that the referenced
* storage survives for as long as referencing pointer Datums can exist.
*
- * Note that just as for struct varatt_external_oid, this struct is stored+ * Note that just as for struct varatt_external_*, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct varatt_indirect
@@ -66,7 +69,7 @@ typedef struct varatt_indirect
* storage. APIs for this, in particular the definition of struct
* ExpandedObjectHeader, are in src/include/utils/expandeddatum.h.
*
- * Note that just as for struct varatt_external_oid, this struct is stored+ * Note that just as for struct varatt_external_*, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct ExpandedObjectHeader ExpandedObjectHeader;
@@ -357,11 +360,18 @@ VARATT_IS_EXTERNAL(const void *PTR)
return VARATT_IS_1B_E(PTR);
}
+/* Is varlena datum a pointer to on-disk toasted data with OID value? */+static inline bool+VARATT_IS_EXTERNAL_ONDISK_OID(const void *PTR)+{+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;+}+
/* Is varlena datum a pointer to on-disk toasted data? */
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;+ return VARATT_IS_EXTERNAL_ONDISK_OID(PTR);
}
/* Is varlena datum an indirect pointer? */
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefileindex e78de312659e..1ef86a245886 100644--- a/src/backend/access/common/Makefile+++ b/src/backend/access/common/Makefile@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
tidstore.o \
toast_compression.o \
+ toast_external.o \
toast_internals.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.cindex 04eedb474c74..6a9b5200203c 100644--- a/src/backend/access/common/detoast.c+++ b/src/backend/access/common/detoast.c@@ -16,6 +16,7 @@
#include "access/detoast.h"
#include "access/table.h"
#include "access/tableam.h"
+#include "access/toast_external.h"
#include "access/toast_internals.h"
#include "common/int.h"
#include "common/pg_lzcompress.h"
@@ -225,12 +226,12 @@ detoast_attr_slice(struct varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_external_info_get_data(attr, &toast_pointer);
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))+ if (!TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer))
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +241,7 @@ detoast_attr_slice(struct varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);+ int32 max_size = toast_pointer.extsize;
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,8 +252,7 @@ detoast_attr_slice(struct varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_GET_COMPRESS_METHOD(toast_pointer) ==- TOAST_PGLZ_COMPRESSION_ID)+ if (toast_pointer.compression_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
/*
@@ -344,20 +344,21 @@ toast_fetch_datum(struct varlena *attr)
{
Relation toastrel;
struct varlena *result;
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;
int32 attrsize;
+ uint64 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
/* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_external_info_get_data(attr, &toast_pointer);- attrsize = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);+ attrsize = toast_pointer.extsize;
result = (struct varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))+ if (TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
else
SET_VARSIZE(result, attrsize + VARHDRSZ);
@@ -365,14 +366,15 @@ toast_fetch_datum(struct varlena *attr)
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
* case. */
+ valueid = toast_pointer.value;
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);+ toastrel = table_open(toast_pointer.toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +400,26 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
{
Relation toastrel;
struct varlena *result;
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;
int32 attrsize;
+ uint64 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
/* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_external_info_get_data(attr, &toast_pointer);++ valueid = toast_pointer.value;
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);+ Assert(!TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);- attrsize = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);+ attrsize = toast_pointer.extsize;
if (sliceoffset >= attrsize)
{
@@ -427,7 +432,7 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer) && slicelength > 0)+ if (TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer) && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +445,7 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))+ if (TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +454,11 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);+ toastrel = table_open(toast_pointer.toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,+ table_relation_fetch_toast_slice(toastrel,+ valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -549,11 +555,7 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- /* va_rawsize is the size of the original datum -- including header */- struct varatt_external_oid toast_pointer;-- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);- result = toast_pointer.va_rawsize;+ result = toast_external_info_get_rawsize(attr);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +612,7 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- struct varatt_external_oid toast_pointer;-- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);- result = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);+ result = toast_external_info_get_extsize(attr);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.buildindex e3cdbe7a22e1..c20f2e88921e 100644--- a/src/backend/access/common/meson.build+++ b/src/backend/access/common/meson.build@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'tidstore.c',
'toast_compression.c',
+ 'toast_external.c',
'toast_internals.c',
'tupconvert.c',
'tupdesc.c',
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.cindex 26aad84d367a..94606a58c8fb 100644--- a/src/backend/access/common/toast_compression.c+++ b/src/backend/access/common/toast_compression.c@@ -19,6 +19,7 @@
#include "access/detoast.h"
#include "access/toast_compression.h"
+#include "access/toast_external.h"
#include "common/pg_lzcompress.h"
#include "varatt.h"
@@ -261,14 +262,7 @@ toast_get_compression_id(struct varlena *attr)
* toast compression header.
*/
if (VARATT_IS_EXTERNAL_ONDISK(attr))
- {- struct varatt_external_oid toast_pointer;-- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);-- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))- cmid = VARATT_EXTERNAL_GET_COMPRESS_METHOD(toast_pointer);- }+ cmid = toast_external_info_get_compression_method(attr);
else if (VARATT_IS_COMPRESSED(attr))
cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr);
diff --git a/src/backend/access/common/toast_external.c b/src/backend/access/common/toast_external.cnew file mode 100644index 000000000000..96ea7be8966e--- /dev/null+++ b/src/backend/access/common/toast_external.c@@ -0,0 +1,161 @@+/*-------------------------------------------------------------------------+ *+ * toast_external.c+ * Functions for the support of external on-disk TOAST pointers.+ *+ * Copyright (c) 2025, PostgreSQL Global Development Group+ *+ *+ * IDENTIFICATION+ * src/backend/access/common/toast_external.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include "access/detoast.h"+#include "access/heaptoast.h"+#include "access/toast_external.h"++/* Callbacks for VARTAG_ONDISK_OID */+static void ondisk_oid_to_external_data(struct varlena *attr,+ toast_external_data *data);+static struct varlena *ondisk_oid_create_external_data(toast_external_data data);+++/*+ * Size of an EXTERNAL datum that contains a standard TOAST pointer (OID+ * value).+ */+#define TOAST_POINTER_OID_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))++/*+ * For now there are only two types, all defined in this file. For now this+ * is the maximum value of vartag_external, which is a historical choice.+ */+#define TOAST_EXTERNAL_INFO_SIZE (VARTAG_ONDISK_OID + 1)++/*+ * The different kinds of on-disk external TOAST pointers, divided by+ * vartag_external.+ *+ * See comments for struct toast_external_info about the details of the+ * individual fields.+ */+static const toast_external_info toast_external_infos[TOAST_EXTERNAL_INFO_SIZE] = {+ [VARTAG_ONDISK_OID] = {+ .toast_pointer_size = TOAST_POINTER_OID_SIZE,+ .maximum_chunk_size = TOAST_MAX_CHUNK_SIZE_OID,+ .to_external_data = ondisk_oid_to_external_data,+ .create_external_data = ondisk_oid_create_external_data,+ },+};+++/* Get toast_external_info of the defined vartag_external */+const toast_external_info *+toast_external_get_info(uint8 tag)+{+ return &toast_external_infos[tag];+}++/*+ * Get external TOAST pointer size based on the attribute type of a TOAST+ * value.+ */+int32+toast_external_info_get_pointer_size(uint8 tag)+{+ return toast_external_infos[tag].toast_pointer_size;+}++/*+ * Assign the vartag_external of a TOAST tuple, based on the TOAST relation+ * it uses and its value.+ *+ * An invalid value can be given by the caller of this routine, in which+ * case a default vartag should be provided based on only the toast relation+ * used.+ */+uint8+toast_external_assign_vartag(Oid toastrelid, uint64 value)+{+ /*+ * If dealing with a code path where a TOAST relation may not be assigned,+ * like heap_toast_insert_or_update(), just use the legacy+ * vartag_external.+ */+ if (!OidIsValid(toastrelid))+ return VARTAG_ONDISK_OID;++ /*+ * Currently there is only one type of vartag_external supported: 4-byte+ * value with OID for the chunk_id type.+ *+ * Note: This routine will be extended to be able to use multiple+ * vartag_external within a single TOAST relation type, that may change+ * depending on the value used.+ */+ return VARTAG_ONDISK_OID;+}++/*+ * Helper routines able to translate the various varatt_external_* from/to+ * the in-memory representation toast_external_data used in the backend.+ */++/* Callbacks for VARTAG_ONDISK_OID */+static void+ondisk_oid_to_external_data(struct varlena *attr, toast_external_data *data)+{+ varatt_external_oid external;++ VARATT_EXTERNAL_GET_POINTER(external, attr);+ data->rawsize = external.va_rawsize;++ /*+ * External size and compression methods are stored in the same field,+ * extract.+ */+ if (VARATT_EXTERNAL_IS_COMPRESSED(external))+ {+ data->extsize = VARATT_EXTERNAL_GET_EXTSIZE(external);+ data->compression_method = VARATT_EXTERNAL_GET_COMPRESS_METHOD(external);+ }+ else+ {+ data->extsize = external.va_extinfo;+ data->compression_method = TOAST_INVALID_COMPRESSION_ID;+ }++ data->value = (uint64) external.va_valueid;+ data->toastrelid = external.va_toastrelid;+}++static struct varlena *+ondisk_oid_create_external_data(toast_external_data data)+{+ struct varlena *result = NULL;+ varatt_external_oid external;++ external.va_rawsize = data.rawsize;++ if (data.compression_method != TOAST_INVALID_COMPRESSION_ID)+ {+ /* Set size and compression method, in a single field. */+ VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(external,+ data.extsize,+ data.compression_method);+ }+ else+ external.va_extinfo = data.extsize;++ external.va_toastrelid = data.toastrelid;+ external.va_valueid = (Oid) data.value;++ result = (struct varlena *) palloc(TOAST_POINTER_OID_SIZE);+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);+ memcpy(VARDATA_EXTERNAL(result), &external, sizeof(external));++ return result;+}diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex d555c02e908e..f659ec41df34 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -18,6 +18,7 @@
#include "access/heapam.h"
#include "access/heaptoast.h"
#include "access/table.h"
+#include "access/toast_external.h"
#include "access/toast_internals.h"
#include "access/xact.h"
#include "catalog/catalog.h"
@@ -127,7 +128,7 @@ toast_save_datum(Relation rel, Datum value,
bool t_isnull[3];
CommandId mycid = GetCurrentCommandId(true);
struct varlena *result;
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;
union
{
struct varlena hdr;
@@ -143,6 +144,8 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ const toast_external_info *info;+ uint8 tag = VARTAG_INDIRECT; /* init value does not matter */
Assert(!VARATT_IS_EXTERNAL(dval));
@@ -174,28 +177,41 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */- toast_pointer.va_extinfo = data_todo;+ toast_pointer.rawsize = data_todo + VARHDRSZ; /* as if not short */+ toast_pointer.extsize = data_todo;++ /*+ * TOAST_INVALID_COMPRESSION_ID means that the varlena is not+ * compressed, see toast_get_compression_id().+ */+ toast_pointer.compression_method = TOAST_INVALID_COMPRESSION_ID;
}
else if (VARATT_IS_COMPRESSED(dval))
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;+ toast_pointer.rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));+ toast_pointer.extsize = data_todo;+ toast_pointer.compression_method = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer));+ Assert(TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);- toast_pointer.va_extinfo = data_todo;+ toast_pointer.rawsize = VARSIZE(dval);+ toast_pointer.extsize = data_todo;++ /*+ * TOAST_INVALID_COMPRESSION_ID means that the varlena is not+ * compressed, see toast_get_compression_id().+ */+ toast_pointer.compression_method = TOAST_INVALID_COMPRESSION_ID;
}
/*
@@ -207,9 +223,9 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;+ toast_pointer.toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);+ toast_pointer.toastrelid = RelationGetRelid(toastrel);
/*
* Choose an OID to use as the value ID for this toast value.
@@ -226,7 +242,7 @@ toast_save_datum(Relation rel, Datum value,
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =+ toast_pointer.value =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
@@ -234,18 +250,18 @@ toast_save_datum(Relation rel, Datum value,
else
{
/* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;+ toast_pointer.value = InvalidOid;
if (oldexternal != NULL)
{
- struct varatt_external_oid old_toast_pointer;+ struct toast_external_data old_toast_pointer;
Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)+ toast_external_info_get_data(oldexternal, &old_toast_pointer);++ if (old_toast_pointer.toastrelid == rel->rd_toastoid)
{
/* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;+ toast_pointer.value = old_toast_pointer.value;
/*
* There is a corner case here: the table rewrite might have
@@ -265,14 +281,14 @@ toast_save_datum(Relation rel, Datum value,
* be reclaimed by VACUUM.
*/
if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))+ toast_pointer.value))
{
/* Match, so short-circuit the data storage loop below */
data_todo = 0;
}
}
}
- if (toast_pointer.va_valueid == InvalidOid)+ if (toast_pointer.value == InvalidOid)
{
/*
* new value; must choose an OID that doesn't conflict in either
@@ -280,24 +296,32 @@ toast_save_datum(Relation rel, Datum value,
*/
do
{
- toast_pointer.va_valueid =+ toast_pointer.value =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));+ toast_pointer.value));
}
}
/*
* Initialize constant parts of the tuple data
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);+ t_values[0] = ObjectIdGetDatum(toast_pointer.value);
t_values[2] = PointerGetDatum(&chunk_data);
t_isnull[0] = false;
t_isnull[1] = false;
t_isnull[2] = false;
+ /*+ * Retrieve the vartag that can be assigned for the new TOAST tuple. This+ * depends on the type of TOAST table and its assigned value.+ */+ tag = toast_external_assign_vartag(toast_pointer.toastrelid,+ toast_pointer.value);+ info = toast_external_get_info(tag);+
/*
* Split up the item into chunks
*/
@@ -310,7 +334,7 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_MAX_CHUNK_SIZE, data_todo);+ chunk_size = Min(info->maximum_chunk_size, data_todo);
/*
* Build a tuple and store it
@@ -368,9 +392,7 @@ toast_save_datum(Relation rel, Datum value,
/*
* Create the TOAST pointer value that we'll return
*/
- result = (struct varlena *) palloc(TOAST_POINTER_SIZE);- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));+ result = info->create_external_data(toast_pointer);
return PointerGetDatum(result);
}
@@ -385,7 +407,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
struct varlena *attr = (struct varlena *) DatumGetPointer(value);
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -398,12 +420,12 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
return;
/* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_external_info_get_data(attr, &toast_pointer);
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);+ toastrel = table_open(toast_pointer.toastrelid, RowExclusiveLock);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -417,7 +439,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));+ ObjectIdGetDatum(toast_pointer.value));
/*
* Find all the chunks. (We don't actually care whether we see them in
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex ae8d502ddcd3..47bf9e84963a 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -28,6 +28,7 @@
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heaptoast.h"
+#include "access/toast_external.h"
#include "access/toast_helper.h"
#include "access/toast_internals.h"
#include "utils/fmgroids.h"
@@ -109,6 +110,7 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
Datum toast_oldvalues[MaxHeapAttributeNumber];
ToastAttrInfo toast_attr[MaxHeapAttributeNumber];
ToastTupleContext ttc;
+ uint8 tag;
/*
* Ignore the INSERT_SPECULATIVE option. Speculative insertions/super
@@ -140,6 +142,16 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
* Prepare for toasting
* ----------
*/
++ /*+ * Retrieve the toast pointer size based on the type of external TOAST+ * pointer assumed to be used.+ */++ /* The default value is invalid, to work as a default. */+ tag = toast_external_assign_vartag(rel->rd_rel->reltoastrelid, InvalidOid);+ ttc.ttc_toast_pointer_size = toast_external_info_get_pointer_size(tag);+
ttc.ttc_rel = rel;
ttc.ttc_values = toast_values;
ttc.ttc_isnull = toast_isnull;
@@ -640,6 +652,8 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ const toast_external_info *info;+ uint8 tag = VARTAG_INDIRECT; /* init value does not matter */
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,7 +661,11 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
- max_chunk_size = TOAST_MAX_CHUNK_SIZE;+ /* Grab the information for toast_external_data */+ tag = toast_external_assign_vartag(RelationGetRelid(toastrel), valueid);+ info = toast_external_get_info(tag);++ max_chunk_size = info->maximum_chunk_size;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.cindex 11f97d65367d..76a7cfe6174e 100644--- a/src/backend/access/table/toast_helper.c+++ b/src/backend/access/table/toast_helper.c@@ -171,8 +171,10 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_POINTER_SIZE);- * if not, no benefit is to be expected by compressing it.+ * The column must have a minimum size of MAXALIGN(tcc_toast_pointer_size);+ * if not, no benefit is to be expected by compressing it. The TOAST+ * pointer size is given by the caller, depending on the type of TOAST+ * table we are dealing with.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -184,10 +186,14 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_POINTER_SIZE);+ int32 biggest_size = 0;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /* Define the lower-bound */+ biggest_size = MAXALIGN(ttc->ttc_toast_pointer_size);+ Assert(biggest_size != 0);+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.cindex 0eb99b4f1377..be33e7de6f8d 100644--- a/src/backend/replication/logical/reorderbuffer.c+++ b/src/backend/replication/logical/reorderbuffer.c@@ -92,6 +92,7 @@
#include "access/detoast.h"
#include "access/heapam.h"
#include "access/rewriteheap.h"
+#include "access/toast_external.h"
#include "access/transam.h"
#include "access/xact.h"
#include "access/xlog_internal.h"
@@ -5102,7 +5103,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
struct varlena *varlena;
/* va_rawsize is the size of the original datum -- including header */
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;
struct varatt_indirect redirect_pointer;
struct varlena *new_datum = NULL;
struct varlena *reconstructed;
@@ -5132,8 +5133,8 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (!VARATT_IS_EXTERNAL(varlena))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena);- toast_valueid = toast_pointer.va_valueid;+ toast_external_info_get_data(varlena, &toast_pointer);+ toast_valueid = toast_pointer.value;
/*
* Check whether the toast tuple changed, replace if so.
@@ -5151,7 +5152,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);+ reconstructed = palloc0(toast_pointer.rawsize);
ent->reconstructed = reconstructed;
@@ -5176,10 +5177,10 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer));+ Assert(data_done == toast_pointer.extsize);
/* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))+ if (TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
else
SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.cindex 38d4e6d45f28..d76386407a08 100644--- a/src/backend/utils/adt/varlena.c+++ b/src/backend/utils/adt/varlena.c@@ -19,6 +19,7 @@
#include "access/detoast.h"
#include "access/toast_compression.h"
+#include "access/toast_external.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_type.h"
#include "common/hashfn.h"
@@ -4219,7 +4220,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
struct varlena *attr;
- struct varatt_external_oid toast_pointer;+ uint64 toast_valueid;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4246,9 +4247,9 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_valueid = toast_external_info_get_value(attr);- PG_RETURN_OID(toast_pointer.va_valueid);+ PG_RETURN_OID(toast_valueid);
}
/*
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 61250799ec07..f3c6cd8860b5 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -415,7 +415,7 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen+most <symbol>TOAST_MAX_CHUNK_SIZE_OID</symbol> bytes (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 030f9fb64b51..11c4507ae6e2 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -16,6 +16,7 @@
#include "access/multixact.h"
#include "access/relation.h"
#include "access/table.h"
+#include "access/toast_external.h"
#include "access/toast_internals.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -73,7 +74,8 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;+ const toast_external_info *info;
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1564,9 +1566,9 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint64 toast_valueid;
int32 max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;+ toast_valueid = ta->toast_pointer.value;- max_chunk_size = TOAST_MAX_CHUNK_SIZE;+ max_chunk_size = ta->info->maximum_chunk_size;
last_chunk_seq = (extsize - 1) / max_chunk_size;
/* Sanity-check the sequence number. */
@@ -1672,7 +1674,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
uint16 infomask;
uint64 toast_pointer_valueid;
CompactAttribute *thisatt;
- struct varatt_external_oid toast_pointer;+ struct toast_external_data toast_pointer;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1774,28 +1776,28 @@ check_tuple_attribute(HeapCheckContext *ctx)
return true;
/* It is external, and we're looking at a page on disk */
- toast_pointer_valueid = toast_pointer.va_valueid;
/*
* Must copy attr into toast_pointer for alignment considerations
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_external_info_get_data(attr, &toast_pointer);+ toast_pointer_valueid = toast_pointer.value;
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)+ if (toast_pointer.rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value %" PRIu64 " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,+ toast_pointer.rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))+ if (TOAST_EXTERNAL_IS_COMPRESSED(toast_pointer))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);+ cmid = toast_pointer.compression_method;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1849,7 +1851,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = (ToastedAttribute *) palloc0(sizeof(ToastedAttribute));
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);+ toast_external_info_get_data(attr, &ta->toast_pointer);+ ta->info = toast_external_get_info(VARTAG_EXTERNAL(attr));
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1876,9 +1879,11 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
uint64 toast_valueid;
- int32 max_chunk_size = TOAST_MAX_CHUNK_SIZE;+ int32 max_chunk_size;- extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);+ extsize = ta->toast_pointer.extsize;++ max_chunk_size = ta->info->maximum_chunk_size;
last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
@@ -1887,7 +1892,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));+ ObjectIdGetDatum(ta->toast_pointer.value));
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1907,7 +1912,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;+ toast_valueid = ta->toast_pointer.value;
if (!found_toasttup)
report_toast_corruption(ctx, ta,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex e6f2e93b2d6f..995dc1f28208 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -4143,6 +4143,8 @@ timeout_params
timerCA
tlist_vinfo
toast_compress_header
+toast_external_data+toast_external_info
tokenize_error_callback_arg
transferMode
transfer_thread_arg
--
2.50.0
v4-0005-Move-static-inline-routines-of-varatt_external_oi.patch (text/x-diff, 4.7K) download
From f91260e4b14d11a9189034aed1edcab81da766e8 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 15:40:04 +0900
Subject: [PATCH v4 05/15] Move static inline routines of varatt_external_oid
to toast_external.c
This isolates most of the knowledge of varatt_external_oid into the
local area where it is manipulated through the toast_external transition
type, with the backend code not requiring it. Extension code should not
need it either, as toast_external should be the layer to use when
looking at external on-dist TOAST varlenas.
---
src/include/varatt.h | 30 -----------------
src/backend/access/common/toast_external.c | 39 ++++++++++++++++++++--
2 files changed, 36 insertions(+), 33 deletions(-)
diff --git a/src/include/varatt.h b/src/include/varatt.hindex b791ce7847ed..631aa2ecc494 100644--- a/src/include/varatt.h+++ b/src/include/varatt.h@@ -513,22 +513,6 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
return ((varattrib_4b *) PTR)->va_compressed.va_tcinfo >> VARLENA_EXTSIZE_BITS;
}
-/*- * Same for external Datums; but note argument is a struct- * varatt_external_oid.- */-static inline Size-VARATT_EXTERNAL_GET_EXTSIZE(struct varatt_external_oid toast_pointer)-{- return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;-}--static inline uint32-VARATT_EXTERNAL_GET_COMPRESS_METHOD(struct varatt_external_oid toast_pointer)-{- return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;-}-
/* Set size and compress method of an externally-stored varlena datum */
/* This has to remain a macro; beware multiple evaluations! */
#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
@@ -539,18 +523,4 @@ VARATT_EXTERNAL_GET_COMPRESS_METHOD(struct varatt_external_oid toast_pointer)
(len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
} while (0)
-/*- * Testing whether an externally-stored value is compressed now requires- * comparing size stored in va_extinfo (the actual length of the external data)- * to rawsize (the original uncompressed datum's size). The latter includes- * VARHDRSZ overhead, the former doesn't. We never use compression unless it- * actually saves space, so we expect either equality or less-than.- */-static inline bool-VARATT_EXTERNAL_IS_COMPRESSED(struct varatt_external_oid toast_pointer)-{- return VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) <- (Size) (toast_pointer.va_rawsize - VARHDRSZ);-}-
#endif
diff --git a/src/backend/access/common/toast_external.c b/src/backend/access/common/toast_external.cindex 96ea7be8966e..79ae02873748 100644--- a/src/backend/access/common/toast_external.c+++ b/src/backend/access/common/toast_external.c@@ -22,6 +22,39 @@ static void ondisk_oid_to_external_data(struct varlena *attr,
toast_external_data *data);
static struct varlena *ondisk_oid_create_external_data(toast_external_data data);
+/*+ * Decompressed size of an on-disk varlena; but note argument is a struct+ * varatt_external_oid.+ */+static inline Size+varatt_external_oid_get_extsize(struct varatt_external_oid toast_pointer)+{+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;+}++/*+ * Compression method of an on-disk varlena; but note argument is a struct+ * varatt_external_oid.+ */+static inline uint32+varatt_external_oid_get_compress_method(struct varatt_external_oid toast_pointer)+{+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;+}++/*+ * Testing whether an externally-stored TOAST value is compressed now requires+ * comparing size stored in va_extinfo (the actual length of the external data)+ * to rawsize (the original uncompressed datum's size). The latter includes+ * VARHDRSZ overhead, the former doesn't. We never use compression unless it+ * actually saves space, so we expect either equality or less-than.+ */+static inline bool+varatt_external_oid_is_compressed(struct varatt_external_oid toast_pointer)+{+ return varatt_external_oid_get_extsize(toast_pointer) <+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);+}
/*
* Size of an EXTERNAL datum that contains a standard TOAST pointer (OID
@@ -117,10 +150,10 @@ ondisk_oid_to_external_data(struct varlena *attr, toast_external_data *data)
* External size and compression methods are stored in the same field,
* extract.
*/
- if (VARATT_EXTERNAL_IS_COMPRESSED(external))+ if (varatt_external_oid_is_compressed(external))
{
- data->extsize = VARATT_EXTERNAL_GET_EXTSIZE(external);- data->compression_method = VARATT_EXTERNAL_GET_COMPRESS_METHOD(external);+ data->extsize = varatt_external_oid_get_extsize(external);+ data->compression_method = varatt_external_oid_get_compress_method(external);
}
else
{
--
2.50.0
v4-0006-Introduce-new-callback-to-get-fresh-TOAST-values.patch (text/x-diff, 6.5K) download
From 3e53ede276101c1eddb5c52a3460c635e67656b4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 15:44:50 +0900
Subject: [PATCH v4 06/15] Introduce new callback to get fresh TOAST values
This callback is called by toast_save_datum() to retrieve a new value
from a source related to the vartag_external we are dealing with. As of
now, it is simply a wrapper around GetNewOidWithIndex() for the "OID"
on-disk TOAST external pointer.
This will be used later on by more external pointer types, like the int8
one.
InvalidToastId is introduced to track the concept of an "invalid" TOAST
value, required for toast_save_datum().
---
src/include/access/toast_external.h | 18 +++++++++++++++
src/backend/access/common/toast_external.c | 11 +++++++++
src/backend/access/common/toast_internals.c | 25 ++++++++++++++-------
src/backend/access/heap/heaptoast.c | 2 +-
4 files changed, 47 insertions(+), 9 deletions(-)
diff --git a/src/include/access/toast_external.h b/src/include/access/toast_external.hindex ad2a10f35c01..9f3177d60e5d 100644--- a/src/include/access/toast_external.h+++ b/src/include/access/toast_external.h@@ -15,9 +15,14 @@
#ifndef TOAST_EXTERNAL_H
#define TOAST_EXTERNAL_H
+#include "access/attnum.h"
#include "access/toast_compression.h"
+#include "utils/relcache.h"
#include "varatt.h"
+/* Invalid TOAST value ID */+#define InvalidToastId 0+
/*
* Intermediate in-memory structure used when creating on-disk
* varatt_external_* or when deserializing varlena contents.
@@ -43,6 +48,7 @@ typedef struct toast_external_data
/*
* Unique ID of value within TOAST table. This could be an OID or an int8
* value. This field is large enough to be able to store any of these.
+ * InvalidToastId if invalid.
*/
uint64 value;
} toast_external_data;
@@ -81,6 +87,18 @@ typedef struct toast_external_info
*/
struct varlena *(*create_external_data) (toast_external_data data);
+ /*+ * Retrieve a new value, to be assigned for a TOAST entry that will be+ * saved. "toastrel" is the relation where the entry is added. "indexid"+ * and "attnum" can be used to check if a value is already in use in the+ * TOAST relation where the new entry is inserted.+ *+ * When "check" is set to true, the value generated should be rechecked+ * with the existing TOAST index.+ */+ uint64 (*get_new_value) (Relation toastrel, Oid indexid,+ AttrNumber attnum);+
} toast_external_info;
/* Retrieve a toast_external_info from a vartag */
diff --git a/src/backend/access/common/toast_external.c b/src/backend/access/common/toast_external.cindex 79ae02873748..5c36e5a11392 100644--- a/src/backend/access/common/toast_external.c+++ b/src/backend/access/common/toast_external.c@@ -16,11 +16,14 @@
#include "access/detoast.h"
#include "access/heaptoast.h"
#include "access/toast_external.h"
+#include "catalog/catalog.h"
/* Callbacks for VARTAG_ONDISK_OID */
static void ondisk_oid_to_external_data(struct varlena *attr,
toast_external_data *data);
static struct varlena *ondisk_oid_create_external_data(toast_external_data data);
+static uint64 ondisk_oid_get_new_value(Relation toastrel, Oid indexid,+ AttrNumber attnum);
/*
* Decompressed size of an on-disk varlena; but note argument is a struct
@@ -81,6 +84,7 @@ static const toast_external_info toast_external_infos[TOAST_EXTERNAL_INFO_SIZE]
.maximum_chunk_size = TOAST_MAX_CHUNK_SIZE_OID,
.to_external_data = ondisk_oid_to_external_data,
.create_external_data = ondisk_oid_create_external_data,
+ .get_new_value = ondisk_oid_get_new_value,
},
};
@@ -192,3 +196,10 @@ ondisk_oid_create_external_data(toast_external_data data)
return result;
}
++static uint64+ondisk_oid_get_new_value(Relation toastrel, Oid indexid,+ AttrNumber attnum)+{+ return GetNewOidWithIndex(toastrel, indexid, attnum);+}diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex f659ec41df34..b0a0ce1e1b9a 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -227,6 +227,15 @@ toast_save_datum(Relation rel, Datum value,
else
toast_pointer.toastrelid = RelationGetRelid(toastrel);
+ /*+ * Retrieve the external TOAST information, with the value still unknown.+ * We need to do this again once we know the actual value assigned, to+ * define the correct vartag_external for the new TOAST tuple.+ */+ tag = toast_external_assign_vartag(toast_pointer.toastrelid,+ InvalidToastId);+ info = toast_external_get_info(tag);+
/*
* Choose an OID to use as the value ID for this toast value.
*
@@ -243,14 +252,14 @@ toast_save_datum(Relation rel, Datum value,
{
/* normal case: just choose an unused OID */
toast_pointer.value =
- GetNewOidWithIndex(toastrel,- RelationGetRelid(toastidxs[validIndex]),- (AttrNumber) 1);+ info->get_new_value(toastrel,+ RelationGetRelid(toastidxs[validIndex]),+ (AttrNumber) 1);
}
else
{
/* rewrite case: check to see if value was in old toast table */
- toast_pointer.value = InvalidOid;+ toast_pointer.value = InvalidToastId;
if (oldexternal != NULL)
{
struct toast_external_data old_toast_pointer;
@@ -288,7 +297,7 @@ toast_save_datum(Relation rel, Datum value,
}
}
}
- if (toast_pointer.value == InvalidOid)+ if (toast_pointer.value == InvalidToastId)
{
/*
* new value; must choose an OID that doesn't conflict in either
@@ -297,9 +306,9 @@ toast_save_datum(Relation rel, Datum value,
do
{
toast_pointer.value =
- GetNewOidWithIndex(toastrel,- RelationGetRelid(toastidxs[validIndex]),- (AttrNumber) 1);+ info->get_new_value(toastrel,+ RelationGetRelid(toastidxs[validIndex]),+ (AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
toast_pointer.value));
}
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex 47bf9e84963a..c215263eb76a 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -149,7 +149,7 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
*/
/* The default value is invalid, to work as a default. */
- tag = toast_external_assign_vartag(rel->rd_rel->reltoastrelid, InvalidOid);+ tag = toast_external_assign_vartag(rel->rd_rel->reltoastrelid, InvalidToastId);
ttc.ttc_toast_pointer_size = toast_external_info_get_pointer_size(tag);
ttc.ttc_rel = rel;
--
2.50.0
v4-0007-Add-catcache-support-for-INT8OID.patch (text/x-diff, 1.3K) download
From d751cd1cb244827f17f9c8752312d3dcd8ee330b Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 18 Jun 2025 16:12:11 +0900
Subject: [PATCH v4 07/15] Add catcache support for INT8OID
This is required to be able to do catalog cache lookups of int8 fields
for toast values of the same type.
---
src/backend/utils/cache/catcache.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.cindex e2cd3feaf81d..55ebf55c7b88 100644--- a/src/backend/utils/cache/catcache.c+++ b/src/backend/utils/cache/catcache.c@@ -240,6 +240,18 @@ int4hashfast(Datum datum)
return murmurhash32((int32) DatumGetInt32(datum));
}
+static bool+int8eqfast(Datum a, Datum b)+{+ return DatumGetInt64(a) == DatumGetInt64(b);+}++static uint32+int8hashfast(Datum datum)+{+ return murmurhash64((int64) DatumGetInt64(datum));+}+
static bool
texteqfast(Datum a, Datum b)
{
@@ -300,6 +312,11 @@ GetCCHashEqFuncs(Oid keytype, CCHashFN *hashfunc, RegProcedure *eqfunc, CCFastEq
*fasteqfunc = int4eqfast;
*eqfunc = F_INT4EQ;
break;
+ case INT8OID:+ *hashfunc = int8hashfast;+ *fasteqfunc = int8eqfast;+ *eqfunc = F_INT8EQ;+ break;
case TEXTOID:
*hashfunc = texthashfast;
*fasteqfunc = texteqfast;
--
2.50.0
v4-0008-Add-GUC-default_toast_type.patch (text/x-diff, 5.4K) download
From 4a8638ee29a1ff85428895831e1290cc6f128a1c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 18 Jun 2025 16:15:19 +0900
Subject: [PATCH v4 08/15] Add GUC default_toast_type
This GUC controls the data type used for newly-created TOAST values,
with two modes supported: "oid" and "int8". This will be used by an
upcoming patch.
---
src/include/access/toast_type.h | 30 +++++++++++++++++++
src/backend/catalog/toasting.c | 4 +++
src/backend/utils/misc/guc_tables.c | 19 ++++++++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
doc/src/sgml/config.sgml | 17 +++++++++++
5 files changed, 71 insertions(+)
create mode 100644 src/include/access/toast_type.h
diff --git a/src/include/access/toast_type.h b/src/include/access/toast_type.hnew file mode 100644index 000000000000..494c2a3e852e--- /dev/null+++ b/src/include/access/toast_type.h@@ -0,0 +1,30 @@+/*-------------------------------------------------------------------------+ *+ * toast_type.h+ * Internal definitions for the types supported by values in TOAST+ * relations.+ *+ * Copyright (c) 2000-2025, PostgreSQL Global Development Group+ *+ * src/include/access/toast_type.h+ *+ *-------------------------------------------------------------------------+ */+#ifndef TOAST_TYPE_H+#define TOAST_TYPE_H++/*+ * GUC support+ *+ * Detault value type in toast table.+ */+extern PGDLLIMPORT int default_toast_type;++typedef enum ToastTypeId+{+ TOAST_TYPE_INVALID = 0,+ TOAST_TYPE_OID = 1,+ TOAST_TYPE_INT8 = 2,+} ToastTypeId;++#endif /* TOAST_TYPE_H */diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.cindex 874a8fc89adb..e595cb61b375 100644--- a/src/backend/catalog/toasting.c+++ b/src/backend/catalog/toasting.c@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/toast_compression.h"
+#include "access/toast_type.h"
#include "access/xact.h"
#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
@@ -33,6 +34,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+/* GUC support */+int default_toast_type = TOAST_TYPE_OID;+
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
LOCKMODE lockmode, bool check,
Oid OIDOldToast);
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.cindex d14b1678e7fe..be523c9ac094 100644--- a/src/backend/utils/misc/guc_tables.c+++ b/src/backend/utils/misc/guc_tables.c@@ -33,6 +33,7 @@
#include "access/gin.h"
#include "access/slru.h"
#include "access/toast_compression.h"
+#include "access/toast_type.h"
#include "access/twophase.h"
#include "access/xlog_internal.h"
#include "access/xlogprefetcher.h"
@@ -464,6 +465,13 @@ static const struct config_enum_entry default_toast_compression_options[] = {
{NULL, 0, false}
};
++static const struct config_enum_entry default_toast_type_options[] = {+ {"oid", TOAST_TYPE_OID, false},+ {"int8", TOAST_TYPE_INT8, false},+ {NULL, 0, false}+};+
static const struct config_enum_entry wal_compression_options[] = {
{"pglz", WAL_COMPRESSION_PGLZ, false},
#ifdef USE_LZ4
@@ -5058,6 +5066,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {+ {"default_toast_type", PGC_USERSET, CLIENT_CONN_STATEMENT,+ gettext_noop("Sets the default type used for TOAST values."),+ NULL+ },+ &default_toast_type,+ TOAST_TYPE_OID,+ default_toast_type_options,+ NULL, NULL, NULL+ },+
{
{"default_transaction_isolation", PGC_USERSET, CLIENT_CONN_STATEMENT,
gettext_noop("Sets the transaction isolation level of each new transaction."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sampleindex a9d8293474af..5f34b14ea39a 100644--- a/src/backend/utils/misc/postgresql.conf.sample+++ b/src/backend/utils/misc/postgresql.conf.sample@@ -753,6 +753,7 @@ autovacuum_worker_slots = 16 # autovacuum worker slots to allocate
#default_table_access_method = 'heap'
#default_tablespace = '' # a tablespace name, '' uses the default
#default_toast_compression = 'pglz' # 'pglz' or 'lz4'
+#default_toast_type = 'oid' # 'oid' or 'int8'
#temp_tablespaces = '' # a list of tablespace names, '' uses
# only default tablespace
#check_function_bodies = on
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgmlindex 20ccb2d6b544..21ccef564f4e 100644--- a/doc/src/sgml/config.sgml+++ b/doc/src/sgml/config.sgml@@ -9834,6 +9834,23 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-default-toast-type" xreflabel="default_toast_type">+ <term><varname>default_toast_type</varname> (<type>enum</type>)+ <indexterm>+ <primary><varname>default_toast_type</varname> configuration parameter</primary>+ </indexterm>+ </term>+ <listitem>+ <para>+ This variable sets the default type for+ <link linkend="storage-toast">TOAST</link> values.+ The value types supported are <literal>oid</literal> and+ <literal>int8</literal>.+ The default is <literal>oid</literal>.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry id="guc-temp-tablespaces" xreflabel="temp_tablespaces">
<term><varname>temp_tablespaces</varname> (<type>string</type>)
<indexterm>
--
2.50.0
v4-0009-Introduce-global-64-bit-TOAST-ID-counter-in-contr.patch (text/x-diff, 21.0K) download
From 41e7d9cbbdaac566cf90e52892361c4139e8bc7a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Jul 2025 09:00:43 +0900
Subject: [PATCH v4 09/15] Introduce global 64-bit TOAST ID counter in control
file
An 8 byte counter is added to the control file, providing a unique
64-bit-wide source for toast value IDs, with the same guarantees as OIDs
in terms of durability. SQL functions and tools looking at the control
file are updated. A WAL record is generated every 8k values generated,
that can be adjusted if required.
Requires a bump of WAL format.
Requires a bump of control file version.
Requires a catalog version bump.
---
src/include/access/toast_counter.h | 34 +++++++
src/include/access/xlog.h | 1 +
src/include/catalog/pg_control.h | 4 +-
src/include/catalog/pg_proc.dat | 6 +-
src/include/storage/lwlocklist.h | 1 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/toast_counter.c | 98 +++++++++++++++++++
src/backend/access/rmgrdesc/xlogdesc.c | 10 ++
src/backend/access/transam/xlog.c | 44 +++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/ipc/ipci.c | 5 +-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/pg_controldata.c | 23 +++--
src/bin/pg_controldata/pg_controldata.c | 2 +
src/bin/pg_resetwal/pg_resetwal.c | 2 +
doc/src/sgml/func/func-info.sgml | 5 +
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 225 insertions(+), 15 deletions(-)
create mode 100644 src/include/access/toast_counter.h
create mode 100644 src/backend/access/common/toast_counter.c
diff --git a/src/include/access/toast_counter.h b/src/include/access/toast_counter.hnew file mode 100644index 000000000000..e2bc79682771--- /dev/null+++ b/src/include/access/toast_counter.h@@ -0,0 +1,34 @@+/*-------------------------------------------------------------------------+ *+ * toast_counter.h+ * Machinery for TOAST value counter.+ *+ * Copyright (c) 2000-2025, PostgreSQL Global Development Group+ *+ * src/include/access/toast_counter.h+ *+ *-------------------------------------------------------------------------+ */+#ifndef TOAST_COUNTER_H+#define TOAST_COUNTER_H++#define FirstToastId 1 /* First TOAST value ID assigned */++/*+ * Structure in shared memory to track TOAST value counter activity.+ * These are protected by ToastIdGenLock.+ */+typedef struct ToastCounterData+{+ uint64 nextId; /* next TOAST value ID to assign */+ uint32 idCount; /* IDs available before WAL work */+} ToastCounterData;++extern PGDLLIMPORT ToastCounterData *ToastCounter;++/* external declarations */+extern Size ToastCounterShmemSize(void);+extern void ToastCounterShmemInit(void);+extern uint64 GetNewToastId(void);++#endif /* TOAST_TYPE_H */diff --git a/src/include/access/xlog.h b/src/include/access/xlog.hindex d12798be3d80..6c5e5feb54bb 100644--- a/src/include/access/xlog.h+++ b/src/include/access/xlog.h@@ -244,6 +244,7 @@ extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
extern void XLogPutNextOid(Oid nextOid);
+extern void XLogPutNextToastId(uint64 nextId);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern void UpdateFullPageWrites(void);
extern void GetFullPageWriteInfo(XLogRecPtr *RedoRecPtr_p, bool *doPageWrites_p);
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.hindex 63e834a6ce47..1194b4928155 100644--- a/src/include/catalog/pg_control.h+++ b/src/include/catalog/pg_control.h@@ -22,7 +22,7 @@
/* Version identifier for this pg_control format */
-#define PG_CONTROL_VERSION 1800+#define PG_CONTROL_VERSION 1900
/* Nonce key length, see below */
#define MOCK_AUTH_NONCE_LEN 32
@@ -45,6 +45,7 @@ typedef struct CheckPoint
Oid nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
+ uint64 nextToastId; /* next free TOAST ID */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
Oid oldestXidDB; /* database with minimum datfrozenxid */
MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */
@@ -80,6 +81,7 @@ typedef struct CheckPoint
/* 0xC0 is used in Postgres 9.5-11 */
#define XLOG_OVERWRITE_CONTRECORD 0xD0
#define XLOG_CHECKPOINT_REDO 0xE0
+#define XLOG_NEXT_TOAST_ID 0xF0
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 118d6da1ace0..1f6df80ad312 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -12318,9 +12318,9 @@
descr => 'pg_controldata checkpoint state information as a function',
proname => 'pg_control_checkpoint', provolatile => 'v',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',- proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time}',+ proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,int8,xid,oid,xid,xid,oid,xid,xid,timestamptz}',+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',+ proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,next_toast_id,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time}',
prosrc => 'pg_control_checkpoint' },
{ oid => '3443',
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.hindex 208d2e3a8ed9..81abc58f0810 100644--- a/src/include/storage/lwlocklist.h+++ b/src/include/storage/lwlocklist.h@@ -85,6 +85,7 @@ PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
+PG_LWLOCK(54, ToastIdGen)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefileindex 1ef86a245886..6e9a3a430c19 100644--- a/src/backend/access/common/Makefile+++ b/src/backend/access/common/Makefile@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
tidstore.o \
toast_compression.o \
+ toast_counter.o \
toast_external.o \
toast_internals.o \
tupconvert.o \
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.buildindex c20f2e88921e..4254132c8dfd 100644--- a/src/backend/access/common/meson.build+++ b/src/backend/access/common/meson.build@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'tidstore.c',
'toast_compression.c',
+ 'toast_counter.c',
'toast_external.c',
'toast_internals.c',
'tupconvert.c',
diff --git a/src/backend/access/common/toast_counter.c b/src/backend/access/common/toast_counter.cnew file mode 100644index 000000000000..94d361d0d5c4--- /dev/null+++ b/src/backend/access/common/toast_counter.c@@ -0,0 +1,98 @@+/*-------------------------------------------------------------------------+ *+ * toast_counter.c+ * Functions for TOAST value counter.+ *+ * Copyright (c) 2025, PostgreSQL Global Development Group+ *+ *+ * IDENTIFICATION+ * src/backend/access/common/toast_counter.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include "access/toast_counter.h"+#include "access/xlog.h"+#include "miscadmin.h"+#include "storage/lwlock.h"+#include "storage/shmem.h"++/* Number of TOAST values to preallocate before WAL work */+#define TOAST_ID_PREFETCH 8192++/* pointer to variables struct in shared memory */+ToastCounterData *ToastCounter = NULL;++/*+ * Initialization of shared memory for ToastCounter.+ */+Size+ToastCounterShmemSize(void)+{+ return sizeof(ToastCounterData);+}++void+ToastCounterShmemInit(void)+{+ bool found;++ /* Initialize shared state struct */+ ToastCounter = ShmemInitStruct("ToastCounter",+ sizeof(ToastCounterData),+ &found);+ if (!IsUnderPostmaster)+ {+ Assert(!found);+ memset(ToastCounter, 0, sizeof(ToastCounterData));+ }+ else+ Assert(found);+}++/*+ * GetNewToastId+ *+ * Toast IDs are generated as a cluster-wide counter. They are 64 bits+ * wide, hence wraparound will unlikely happen.+ */+uint64+GetNewToastId(void)+{+ uint64 result;++ if (RecoveryInProgress())+ elog(ERROR, "cannot assign TOAST IDs during recovery");++ LWLockAcquire(ToastIdGenLock, LW_EXCLUSIVE);++ /*+ * Check for initialization or wraparound of the toast counter ID.+ * InvalidToastId (0) should never be returned. We are 64 bit-wide, hence+ * wraparound is unlikely going to happen, but this check is cheap so+ * let's play it safe.+ */+ if (ToastCounter->nextId < ((uint64) FirstToastId))+ {+ /* Most-likely first bootstrap or initdb assignment */+ ToastCounter->nextId = FirstToastId;+ ToastCounter->idCount = 0;+ }++ /* If running out of logged for TOAST IDs, log more */+ if (ToastCounter->idCount == 0)+ {+ XLogPutNextToastId(ToastCounter->nextId + TOAST_ID_PREFETCH);+ ToastCounter->idCount = TOAST_ID_PREFETCH;+ }++ result = ToastCounter->nextId;+ (ToastCounter->nextId)++;+ (ToastCounter->idCount)--;++ LWLockRelease(ToastIdGenLock);++ return result;+}diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.cindex cd6c2a2f650a..6786af4064b8 100644--- a/src/backend/access/rmgrdesc/xlogdesc.c+++ b/src/backend/access/rmgrdesc/xlogdesc.c@@ -96,6 +96,13 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
memcpy(&nextOid, rec, sizeof(Oid));
appendStringInfo(buf, "%u", nextOid);
}
+ else if (info == XLOG_NEXT_TOAST_ID)+ {+ uint64 nextId;++ memcpy(&nextId, rec, sizeof(uint64));+ appendStringInfo(buf, "%" PRIu64, nextId);+ }
else if (info == XLOG_RESTORE_POINT)
{
xl_restore_point *xlrec = (xl_restore_point *) rec;
@@ -218,6 +225,9 @@ xlog_identify(uint8 info)
case XLOG_CHECKPOINT_REDO:
id = "CHECKPOINT_REDO";
break;
+ case XLOG_NEXT_TOAST_ID:+ id = "NEXT_TOAST_ID";+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 9a4de1616bcc..172ee27fe4cf 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -53,6 +53,7 @@
#include "access/rewriteheap.h"
#include "access/subtrans.h"
#include "access/timeline.h"
+#include "access/toast_counter.h"
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xact.h"
@@ -5259,6 +5260,7 @@ BootStrapXLOG(uint32 data_checksum_version)
checkPoint.nextOid = FirstGenbkiObjectId;
checkPoint.nextMulti = FirstMultiXactId;
checkPoint.nextMultiOffset = 0;
+ checkPoint.nextToastId = FirstToastId;
checkPoint.oldestXid = FirstNormalTransactionId;
checkPoint.oldestXidDB = Template1DbOid;
checkPoint.oldestMulti = FirstMultiXactId;
@@ -5271,6 +5273,10 @@ BootStrapXLOG(uint32 data_checksum_version)
TransamVariables->nextXid = checkPoint.nextXid;
TransamVariables->nextOid = checkPoint.nextOid;
TransamVariables->oidCount = 0;
++ ToastCounter->nextId = checkPoint.nextToastId;+ ToastCounter->idCount = 0;+
MultiXactSetNextMXact(checkPoint.nextMulti, checkPoint.nextMultiOffset);
AdvanceOldestClogXid(checkPoint.oldestXid);
SetTransactionIdLimit(checkPoint.oldestXid, checkPoint.oldestXidDB);
@@ -5752,6 +5758,8 @@ StartupXLOG(void)
TransamVariables->nextXid = checkPoint.nextXid;
TransamVariables->nextOid = checkPoint.nextOid;
TransamVariables->oidCount = 0;
+ ToastCounter->nextId = checkPoint.nextToastId;+ ToastCounter->idCount = 0;
MultiXactSetNextMXact(checkPoint.nextMulti, checkPoint.nextMultiOffset);
AdvanceOldestClogXid(checkPoint.oldestXid);
SetTransactionIdLimit(checkPoint.oldestXid, checkPoint.oldestXidDB);
@@ -7299,6 +7307,12 @@ CreateCheckPoint(int flags)
checkPoint.nextOid += TransamVariables->oidCount;
LWLockRelease(OidGenLock);
+ LWLockAcquire(ToastIdGenLock, LW_SHARED);+ checkPoint.nextToastId = ToastCounter->nextId;+ if (!shutdown)+ checkPoint.nextToastId += ToastCounter->idCount;+ LWLockRelease(ToastIdGenLock);+
MultiXactGetCheckptMulti(shutdown,
&checkPoint.nextMulti,
&checkPoint.nextMultiOffset,
@@ -8234,6 +8248,22 @@ XLogPutNextOid(Oid nextOid)
*/
}
+/*+ * Write a NEXT_TOAST_ID log record.+ */+void+XLogPutNextToastId(uint64 nextId)+{+ XLogBeginInsert();+ XLogRegisterData(&nextId, sizeof(uint64));+ (void) XLogInsert(RM_XLOG_ID, XLOG_NEXT_TOAST_ID);++ /*+ * The next TOAST value ID is not flushed immediately, for the same reason+ * as above for the OIDs in XLogPutNextOid().+ */+}+
/*
* Write an XLOG SWITCH record.
*
@@ -8449,6 +8479,16 @@ xlog_redo(XLogReaderState *record)
TransamVariables->oidCount = 0;
LWLockRelease(OidGenLock);
}
+ else if (info == XLOG_NEXT_TOAST_ID)+ {+ uint64 nextToastId;++ memcpy(&nextToastId, XLogRecGetData(record), sizeof(uint64));+ LWLockAcquire(ToastIdGenLock, LW_EXCLUSIVE);+ ToastCounter->nextId = nextToastId;+ ToastCounter->idCount = 0;+ LWLockRelease(ToastIdGenLock);+ }
else if (info == XLOG_CHECKPOINT_SHUTDOWN)
{
CheckPoint checkPoint;
@@ -8463,6 +8503,10 @@ xlog_redo(XLogReaderState *record)
TransamVariables->nextOid = checkPoint.nextOid;
TransamVariables->oidCount = 0;
LWLockRelease(OidGenLock);
+ LWLockAcquire(ToastIdGenLock, LW_EXCLUSIVE);+ ToastCounter->nextId = checkPoint.nextToastId;+ ToastCounter->idCount = 0;+ LWLockRelease(ToastIdGenLock);
MultiXactSetNextMXact(checkPoint.nextMulti,
checkPoint.nextMultiOffset);
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.cindex cc03f0706e9c..bb0337d37201 100644--- a/src/backend/replication/logical/decode.c+++ b/src/backend/replication/logical/decode.c@@ -188,6 +188,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPI:
case XLOG_OVERWRITE_CONTRECORD:
case XLOG_CHECKPOINT_REDO:
+ case XLOG_NEXT_TOAST_ID:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.cindex 2fa045e6b0f6..9102c267d7b0 100644--- a/src/backend/storage/ipc/ipci.c+++ b/src/backend/storage/ipc/ipci.c@@ -20,6 +20,7 @@
#include "access/nbtree.h"
#include "access/subtrans.h"
#include "access/syncscan.h"
+#include "access/toast_counter.h"
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogprefetcher.h"
@@ -119,6 +120,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, ProcGlobalShmemSize());
size = add_size(size, XLogPrefetchShmemSize());
size = add_size(size, VarsupShmemSize());
+ size = add_size(size, ToastCounterShmemSize());
size = add_size(size, XLOGShmemSize());
size = add_size(size, XLogRecoveryShmemSize());
size = add_size(size, CLOGShmemSize());
@@ -280,8 +282,9 @@ CreateOrAttachShmemStructs(void)
DSMRegistryShmemInit();
/*
- * Set up xlog, clog, and buffers+ * Set up TOAST counter, xlog, clog, and buffers
*/
+ ToastCounterShmemInit();
VarsupShmemInit();
XLOGShmemInit();
XLogPrefetchShmemInit();
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txtindex 0be307d2ca04..a36969ac6659 100644--- a/src/backend/utils/activity/wait_event_names.txt+++ b/src/backend/utils/activity/wait_event_names.txt@@ -352,6 +352,7 @@ DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
+ToastIdGen "Waiting to allocate a new TOAST value ID."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.cindex 6d036e3bf328..e4abf8593b8d 100644--- a/src/backend/utils/misc/pg_controldata.c+++ b/src/backend/utils/misc/pg_controldata.c@@ -69,8 +69,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
- Datum values[18];- bool nulls[18];+ Datum values[19];+ bool nulls[19];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -130,30 +130,33 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
values[9] = TransactionIdGetDatum(ControlFile->checkPointCopy.nextMultiOffset);
nulls[9] = false;
- values[10] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestXid);+ values[10] = UInt64GetDatum(ControlFile->checkPointCopy.nextToastId);
nulls[10] = false;
- values[11] = ObjectIdGetDatum(ControlFile->checkPointCopy.oldestXidDB);+ values[11] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestXid);
nulls[11] = false;
- values[12] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestActiveXid);+ values[12] = ObjectIdGetDatum(ControlFile->checkPointCopy.oldestXidDB);
nulls[12] = false;
- values[13] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestMulti);+ values[13] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestActiveXid);
nulls[13] = false;
- values[14] = ObjectIdGetDatum(ControlFile->checkPointCopy.oldestMultiDB);+ values[14] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestMulti);
nulls[14] = false;
- values[15] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestCommitTsXid);+ values[15] = ObjectIdGetDatum(ControlFile->checkPointCopy.oldestMultiDB);
nulls[15] = false;
- values[16] = TransactionIdGetDatum(ControlFile->checkPointCopy.newestCommitTsXid);+ values[16] = TransactionIdGetDatum(ControlFile->checkPointCopy.oldestCommitTsXid);
nulls[16] = false;
- values[17] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));+ values[17] = TransactionIdGetDatum(ControlFile->checkPointCopy.newestCommitTsXid);
nulls[17] = false;
+ values[18] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));+ nulls[18] = false;+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.cindex 10de058ce91f..99200262b57c 100644--- a/src/bin/pg_controldata/pg_controldata.c+++ b/src/bin/pg_controldata/pg_controldata.c@@ -266,6 +266,8 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.nextMulti);
printf(_("Latest checkpoint's NextMultiOffset: %u\n"),
ControlFile->checkPointCopy.nextMultiOffset);
+ printf(_("Latest checkpoint's NextToastID: %" PRIu64 "\n"),+ ControlFile->checkPointCopy.nextToastId);
printf(_("Latest checkpoint's oldestXID: %u\n"),
ControlFile->checkPointCopy.oldestXid);
printf(_("Latest checkpoint's oldestXID's DB: %u\n"),
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.cindex e876f35f38ed..bb324c710911 100644--- a/src/bin/pg_resetwal/pg_resetwal.c+++ b/src/bin/pg_resetwal/pg_resetwal.c@@ -45,6 +45,7 @@
#include "access/heaptoast.h"
#include "access/multixact.h"
+#include "access/toast_counter.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xlog_internal.h"
@@ -686,6 +687,7 @@ GuessControlValues(void)
ControlFile.checkPointCopy.nextOid = FirstGenbkiObjectId;
ControlFile.checkPointCopy.nextMulti = FirstMultiXactId;
ControlFile.checkPointCopy.nextMultiOffset = 0;
+ ControlFile.checkPointCopy.nextToastId = FirstToastId;
ControlFile.checkPointCopy.oldestXid = FirstNormalTransactionId;
ControlFile.checkPointCopy.oldestXidDB = InvalidOid;
ControlFile.checkPointCopy.oldestMulti = FirstMultiXactId;
diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgmlindex b507bfaf64b1..9d7548825e5a 100644--- a/doc/src/sgml/func/func-info.sgml+++ b/doc/src/sgml/func/func-info.sgml@@ -3395,6 +3395,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
<entry><type>xid</type></entry>
</row>
+ <row>+ <entry><structfield>next_toast_id</structfield></entry>+ <entry><type>bigint</type></entry>+ </row>+
<row>
<entry><structfield>oldest_xid</structfield></entry>
<entry><type>xid</type></entry>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 995dc1f28208..d6bf4e47991b 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -3055,6 +3055,7 @@ TmFromChar
TmToChar
ToastAttrInfo
ToastCompressionId
+ToastCounterData
ToastTupleContext
ToastedAttribute
TocEntry
--
2.50.0
v4-0010-Switch-pg_column_toast_chunk_id-return-value-from.patch (text/x-diff, 2.3K) download
From f841b4bbafe5203fd9edc90b6fd8ad049ea253a0 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 15:52:05 +0900
Subject: [PATCH v4 10/15] Switch pg_column_toast_chunk_id() return value from
oid to bigint
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 1f6df80ad312..27887436ede0 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -7735,7 +7735,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',+ prorettype => 'int8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.cindex d76386407a08..26c720449f7b 100644--- a/src/backend/utils/adt/varlena.c+++ b/src/backend/utils/adt/varlena.c@@ -4249,7 +4249,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
toast_valueid = toast_external_info_get_value(attr);
- PG_RETURN_OID(toast_valueid);+ PG_RETURN_UINT64(toast_valueid);
}
/*
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgmlindex 446fdfe56f4f..1e83584ac579 100644--- a/doc/src/sgml/func/func-admin.sgml+++ b/doc/src/sgml/func/func-admin.sgml@@ -1571,7 +1571,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>+ <returnvalue>bigint</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.50.0
v4-0011-Add-support-for-bigint-TOAST-values.patch (text/x-diff, 13.7K) download
From 27275a6a55857aa7543c4571d3315c6ad462c15c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 1 Aug 2025 17:17:51 +0900
Subject: [PATCH v4 11/15] Add support for bigint TOAST values
This commit adds the possibility to define TOAST tables with bigint as
value ID, based on the GUC default_toast_type. All the external TOAST
pointers still rely on varatt_external and a single vartag, with all the
values inserted in the bigint TOAST tables fed from the existing OID
value generator. This will be changed in an upcoming patch that adds
more vartag_external types and its associated structures, with the code
being able to use a different external TOAST pointer depending on the
attribute type of chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types supported.
XXX: Catalog version bump required.
---
src/backend/access/common/toast_internals.c | 67 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 20 ++++--
src/backend/catalog/toasting.c | 32 +++++++++-
doc/src/sgml/storage.sgml | 7 ++-
contrib/amcheck/verify_heapam.c | 19 ++++--
5 files changed, 115 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex b0a0ce1e1b9a..98a240242127 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -18,6 +18,7 @@
#include "access/heapam.h"
#include "access/heaptoast.h"
#include "access/table.h"
+#include "access/toast_counter.h"
#include "access/toast_external.h"
#include "access/toast_internals.h"
#include "access/xact.h"
@@ -26,6 +27,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, uint64 valueid);
static bool toastid_valueid_exists(Oid toastrelid, uint64 valueid);
@@ -146,8 +148,10 @@ toast_save_datum(Relation rel, Datum value,
int validIndex;
const toast_external_info *info;
uint8 tag = VARTAG_INDIRECT; /* init value does not matter */
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -237,20 +241,23 @@ toast_save_datum(Relation rel, Datum value,
info = toast_external_get_info(tag);
/*
- * Choose an OID to use as the value ID for this toast value.+ * Choose a new value to use as the value ID for this toast value, be it+ * for OID or int8-based TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.+ * conflict with either new or existing toast value IDs. If the TOAST+ * table uses 8-byte value IDs, we should not really care much about+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
- /* normal case: just choose an unused OID */+ /* normal case: just choose an unused ID */
toast_pointer.value =
info->get_new_value(toastrel,
RelationGetRelid(toastidxs[validIndex]),
@@ -269,7 +276,7 @@ toast_save_datum(Relation rel, Datum value,
if (old_toast_pointer.toastrelid == rel->rd_toastoid)
{
- /* This value came from the old toast table; reuse its OID */+ /* This value came from the old toast table; reuse its ID */
toast_pointer.value = old_toast_pointer.value;
/*
@@ -300,8 +307,8 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.value == InvalidToastId)
{
/*
- * new value; must choose an OID that doesn't conflict in either- * old or new toast table+ * new value; must choose a value that doesn't conflict in either+ * old or new toast table.
*/
do
{
@@ -317,7 +324,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Initialize constant parts of the tuple data
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.value);+ if (toast_typid == OIDOID)+ t_values[0] = ObjectIdGetDatum(toast_pointer.value);+ else if (toast_typid == INT8OID)+ t_values[0] = Int64GetDatum(toast_pointer.value);
t_values[2] = PointerGetDatum(&chunk_data);
t_isnull[0] = false;
t_isnull[1] = false;
@@ -424,6 +434,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -435,6 +446,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;+ Assert(toast_typid == OIDOID || toast_typid == INT8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -445,10 +458,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(toast_pointer.value));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(toast_pointer.value));+ else if (toast_typid == INT8OID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_INT8EQ,+ Int64GetDatum(toast_pointer.value));+ else+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -495,6 +516,7 @@ toastrel_valueid_exists(Relation toastrel, uint64 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -502,13 +524,24 @@ toastrel_valueid_exists(Relation toastrel, uint64 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;+ Assert(toast_typid == OIDOID || toast_typid == INT8OID);+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(valueid));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(valueid));+ else if (toast_typid == INT8OID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_INT8EQ,+ Int64GetDatum(valueid));+ else+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex c215263eb76a..a01c1d627c6f 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -654,6 +654,7 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
int32 max_chunk_size;
const toast_external_info *info;
uint8 tag = VARTAG_INDIRECT; /* init value does not matter */
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -667,16 +668,27 @@ heap_fetch_toast_slice(Relation toastrel, uint64 valueid, int32 attrsize,
max_chunk_size = info->maximum_chunk_size;
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;+ Assert(toast_typid == OIDOID || toast_typid == INT8OID);+
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
endchunk = (sliceoffset + slicelength - 1) / max_chunk_size;
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(valueid));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey[0],+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(valueid));+ else if (toast_typid == INT8OID)+ ScanKeyInit(&toastkey[0],+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_INT8EQ,+ Int64GetDatum(valueid));+ else+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.cindex e595cb61b375..27295866c490 100644--- a/src/backend/catalog/toasting.c+++ b/src/backend/catalog/toasting.c@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
/* GUC support */
@@ -149,6 +150,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_typid = InvalidOid;
/*
* Is it already toasted?
@@ -204,11 +206,34 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*+ * Determine the type OID to use for the value. If OIDOldToast is+ * defined, we need to rely on the existing table for the job because+ * we do not want to create an inconsistent relation that would conflict+ * with the parent and break the world.+ */+ if (!OidIsValid(OIDOldToast))+ {+ if (default_toast_type == TOAST_TYPE_OID)+ toast_typid = OIDOID;+ else if (default_toast_type == TOAST_TYPE_INT8)+ toast_typid = INT8OID;+ else+ Assert(false);+ }+ else+ {+ /* For the chunk_id type */+ toast_typid = get_atttype(OIDOldToast, 1);+ if (!OidIsValid(toast_typid))+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);+ }+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,+ toast_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
@@ -316,7 +341,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;+ if (toast_typid == OIDOID)+ opclassIds[0] = OID_BTREE_OPS_OID;+ else if (toast_typid == INT8OID)+ opclassIds[0] = INT8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex f3c6cd8860b5..564783a1c559 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -419,14 +419,15 @@ most <symbol>TOAST_MAX_CHUNK_SIZE_OID</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID-identifying the particular <acronym>TOAST</acronym>ed value),+<acronym>TOAST</acronym> table has the columns+<structfield>chunk_id</structfield> (an OID or an 8-byte integer identifying+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 11c4507ae6e2..833811c75437 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -1880,6 +1880,9 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
uint64 toast_valueid;
int32 max_chunk_size;
+ Oid toast_typid;++ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
extsize = ta->toast_pointer.extsize;
@@ -1889,10 +1892,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(ta->toast_pointer.value));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(ta->toast_pointer.value));+ else if (toast_typid == INT8OID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_INT8EQ,+ Int64GetDatum(ta->toast_pointer.value));+ else+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.50.0
v4-0012-Add-tests-for-TOAST-relations-with-bigint-as-valu.patch (text/x-diff, 19.4K) download
From 63c32ec1f0d50b2f1278418b38269bd8a3410666 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 19 Jun 2025 11:15:48 +0900
Subject: [PATCH v4 12/15] Add tests for TOAST relations with bigint as value
type
This adds coverage for relations created with default_toast_type =
'int8', for external TOAST pointers both compressed and uncompressed.
---
src/test/regress/expected/strings.out | 238 ++++++++++++++++++++++----
src/test/regress/sql/strings.sql | 142 +++++++++++----
2 files changed, 305 insertions(+), 75 deletions(-)
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.outindex ba302da51e7b..9b881b4b7c57 100644--- a/src/test/regress/expected/strings.out+++ b/src/test/regress/expected/strings.out@@ -1945,21 +1945,40 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values+-- test substr with toasted text values, for all types of TOAST relations+-- supported.
--
-CREATE TABLE toasttest(f1 text);-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+SET default_toast_type = 'oid';+CREATE TABLE toasttest_oid(f1 text);+SET default_toast_type = 'int8';+CREATE TABLE toasttest_int8(f1 text);+RESET default_toast_type;+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_int8 values(repeat('1234567890',10000));+insert into toasttest_int8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+alter table toasttest_int8 alter column f1 set storage external;+insert into toasttest_int8 values(repeat('1234567890',10000));+insert into toasttest_int8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+ substr +--------+ 123+ 123+ 123+ 123+(4 rows)++SELECT substr(f1, -1, 5) from toasttest_int8;
substr
--------
123
@@ -1969,11 +1988,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+ERROR: negative substring length not allowed+SELECT substr(f1, 5, -1) from toasttest_int8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995) from toasttest_int8;
substr
--------
567890
@@ -1984,7 +2014,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -1993,50 +2023,108 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+SELECT substr(f1, 99995, 10) from toasttest_int8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++-- TRUNCATE cases for TOAST relations with OID values.+TRUNCATE TABLE toasttest_oid;+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;-ALTER TABLE toasttest set (toast_tuple_target = 4080);-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+TRUNCATE TABLE toasttest_oid;+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;+DROP TABLE toasttest_oid;+-- TRUNCATE cases for TOAST relation with int8 values.+TRUNCATE TABLE toasttest_int8;+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+-- expect >0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_int8';+ is_empty +----------+ f+(1 row)++TRUNCATE TABLE toasttest_int8;+ALTER TABLE toasttest_int8 set (toast_tuple_target = 4080);+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+-- expect 0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_int8';+ is_empty +----------+ t+(1 row)++DROP TABLE toasttest_int8;
--
--- test substr with toasted bytea values+-- test substr with toasted bytea values, for all types of TOAST relations+-- supported.
--
-CREATE TABLE toasttest(f1 bytea);-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+SET default_toast_type = 'oid';+CREATE TABLE toasttest_oid(f1 bytea);+SET default_toast_type = 'int8';+CREATE TABLE toasttest_int8(f1 bytea);+RESET default_toast_type;+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_int8 alter column f1 set storage external;+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+ substr +--------+ 123+ 123+ 123+ 123+(4 rows)++SELECT substr(f1, -1, 5) from toasttest_int8;
substr
--------
123
@@ -2046,11 +2134,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+ERROR: negative substring length not allowed+SELECT substr(f1, 5, -1) from toasttest_int8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995) from toasttest_int8;
substr
--------
567890
@@ -2061,7 +2160,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2070,7 +2169,72 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;+SELECT substr(f1, 99995, 10) from toasttest_int8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++-- A relation rewrite leaves the TOAST value attributes unchanged.+VACUUM FULL toasttest_oid;+VACUUM FULL toasttest_int8;+SELECT c1.relname, a.atttypid::regtype+ FROM pg_attribute AS a,+ pg_class AS c1,+ pg_class AS c2+ WHERE+ c1.relname IN ('toasttest_oid', 'toasttest_int8') AND+ c1.reltoastrelid = c2.oid AND+ a.attrelid = c2.oid AND+ a.attname = 'chunk_id'+ ORDER BY c1.relname COLLATE "C";+ relname | atttypid +----------------+----------+ toasttest_int8 | bigint+ toasttest_oid | oid+(2 rows)++-- Check that data slices are still accessible.+SELECT substr(f1, 99995) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995) from toasttest_int8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995, 10) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995, 10) from toasttest_int8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++DROP TABLE toasttest_oid, toasttest_int8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sqlindex b94004cc08ce..6dc2bbfed70a 100644--- a/src/test/regress/sql/strings.sql+++ b/src/test/regress/sql/strings.sql@@ -553,89 +553,155 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values+-- test substr with toasted text values, for all types of TOAST relations+-- supported.
--
-CREATE TABLE toasttest(f1 text);+SET default_toast_type = 'oid';+CREATE TABLE toasttest_oid(f1 text);+SET default_toast_type = 'int8';+CREATE TABLE toasttest_int8(f1 text);+RESET default_toast_type;-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_int8 values(repeat('1234567890',10000));+insert into toasttest_int8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+alter table toasttest_int8 alter column f1 set storage external;+insert into toasttest_int8 values(repeat('1234567890',10000));+insert into toasttest_int8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+SELECT substr(f1, -1, 5) from toasttest_int8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+SELECT substr(f1, 5, -1) from toasttest_int8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+SELECT substr(f1, 99995) from toasttest_int8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;+SELECT substr(f1, 99995, 10) from toasttest_int8;-TRUNCATE TABLE toasttest;-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+-- TRUNCATE cases for TOAST relations with OID values.+TRUNCATE TABLE toasttest_oid;+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';--TRUNCATE TABLE toasttest;-ALTER TABLE toasttest set (toast_tuple_target = 4080);-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+ FROM pg_class where relname = 'toasttest_oid';+TRUNCATE TABLE toasttest_oid;+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';+ FROM pg_class where relname = 'toasttest_oid';+DROP TABLE toasttest_oid;-DROP TABLE toasttest;+-- TRUNCATE cases for TOAST relation with int8 values.+TRUNCATE TABLE toasttest_int8;+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+-- expect >0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_int8';+TRUNCATE TABLE toasttest_int8;+ALTER TABLE toasttest_int8 set (toast_tuple_target = 4080);+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+INSERT INTO toasttest_int8 values (repeat('1234567890',300));+-- expect 0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_int8';+DROP TABLE toasttest_int8;
--
--- test substr with toasted bytea values+-- test substr with toasted bytea values, for all types of TOAST relations+-- supported.
--
-CREATE TABLE toasttest(f1 bytea);+SET default_toast_type = 'oid';+CREATE TABLE toasttest_oid(f1 bytea);+SET default_toast_type = 'int8';+CREATE TABLE toasttest_int8(f1 bytea);+RESET default_toast_type;-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_int8 alter column f1 set storage external;+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_int8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+SELECT substr(f1, -1, 5) from toasttest_int8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+SELECT substr(f1, 5, -1) from toasttest_int8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+SELECT substr(f1, 99995) from toasttest_int8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;+SELECT substr(f1, 99995, 10) from toasttest_int8;-DROP TABLE toasttest;+-- A relation rewrite leaves the TOAST value attributes unchanged.+VACUUM FULL toasttest_oid;+VACUUM FULL toasttest_int8;+SELECT c1.relname, a.atttypid::regtype+ FROM pg_attribute AS a,+ pg_class AS c1,+ pg_class AS c2+ WHERE+ c1.relname IN ('toasttest_oid', 'toasttest_int8') AND+ c1.reltoastrelid = c2.oid AND+ a.attrelid = c2.oid AND+ a.attname = 'chunk_id'+ ORDER BY c1.relname COLLATE "C";+-- Check that data slices are still accessible.+SELECT substr(f1, 99995) from toasttest_oid;+SELECT substr(f1, 99995) from toasttest_int8;+SELECT substr(f1, 99995, 10) from toasttest_oid;+SELECT substr(f1, 99995, 10) from toasttest_int8;++DROP TABLE toasttest_oid, toasttest_int8;
-- test internally compressing datums
--
2.50.0
v4-0013-Add-support-for-TOAST-table-types-in-pg_dump-and-.patch (text/x-diff, 21.3K) download
From 88c8491844ef525d23c46cf9b290836dfb5db0da Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 16:04:54 +0900
Subject: [PATCH v4 13/15] Add support for TOAST table types in pg_dump and
pg_restore
This includes the possibility to perform binary upgrades with TOAST
table types applied to a new cluster, relying on SET commands based on
default_toast_type to apply one type of TOAST table or the other.
Some tests are included, this is a pretty mechanical change.
Dump format is bumped to 1.17 due to the addition of the TOAST table
type in the custom format.
---
src/bin/pg_dump/pg_backup.h | 2 +
src/bin/pg_dump/pg_backup_archiver.c | 69 +++++++++++++++++++++++++++-
src/bin/pg_dump/pg_backup_archiver.h | 6 ++-
src/bin/pg_dump/pg_dump.c | 21 +++++++++
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/pg_dumpall.c | 5 ++
src/bin/pg_dump/pg_restore.c | 4 ++
src/bin/pg_dump/t/002_pg_dump.pl | 35 ++++++++++++++
doc/src/sgml/ref/pg_dump.sgml | 12 +++++
doc/src/sgml/ref/pg_dumpall.sgml | 12 +++++
doc/src/sgml/ref/pg_restore.sgml | 12 +++++
11 files changed, 177 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.hindex 4ebef1e86445..f99d5f0d3d1b 100644--- a/src/bin/pg_dump/pg_backup.h+++ b/src/bin/pg_dump/pg_backup.h@@ -99,6 +99,7 @@ typedef struct _restoreOptions
int noOwner; /* Don't try to match original object owner */
int noTableAm; /* Don't issue table-AM-related commands */
int noTablespace; /* Don't issue tablespace-related commands */
+ int noToastType; /* Don't issue TOAST-type-related commands */
int disable_triggers; /* disable triggers during data-only
* restore */
int use_setsessauth; /* Use SET SESSION AUTHORIZATION commands
@@ -192,6 +193,7 @@ typedef struct _dumpOptions
int disable_triggers;
int outputNoTableAm;
int outputNoTablespaces;
+ int outputNoToastType;
int use_setsessauth;
int enable_row_security;
int load_via_partition_root;
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.cindex dce88f040ace..5690c0850559 100644--- a/src/bin/pg_dump/pg_backup_archiver.c+++ b/src/bin/pg_dump/pg_backup_archiver.c@@ -185,6 +185,7 @@ dumpOptionsFromRestoreOptions(RestoreOptions *ropt)
dopt->outputNoOwner = ropt->noOwner;
dopt->outputNoTableAm = ropt->noTableAm;
dopt->outputNoTablespaces = ropt->noTablespace;
+ dopt->outputNoToastType = ropt->noToastType;
dopt->disable_triggers = ropt->disable_triggers;
dopt->use_setsessauth = ropt->use_setsessauth;
dopt->disable_dollar_quoting = ropt->disable_dollar_quoting;
@@ -1244,6 +1245,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
newToc->namespace = opts->namespace ? pg_strdup(opts->namespace) : NULL;
newToc->tablespace = opts->tablespace ? pg_strdup(opts->tablespace) : NULL;
newToc->tableam = opts->tableam ? pg_strdup(opts->tableam) : NULL;
+ newToc->toasttype = opts->toasttype ? pg_strdup(opts->toasttype) : NULL;
newToc->relkind = opts->relkind;
newToc->owner = opts->owner ? pg_strdup(opts->owner) : NULL;
newToc->desc = pg_strdup(opts->description);
@@ -2403,6 +2405,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
AH->currSchema = NULL; /* ditto */
AH->currTablespace = NULL; /* ditto */
AH->currTableAm = NULL; /* ditto */
+ AH->currToastType = NULL; /* ditto */
AH->toc = (TocEntry *) pg_malloc0(sizeof(TocEntry));
@@ -2670,6 +2673,7 @@ WriteToc(ArchiveHandle *AH)
WriteStr(AH, te->tablespace);
WriteStr(AH, te->tableam);
WriteInt(AH, te->relkind);
+ WriteStr(AH, te->toasttype);
WriteStr(AH, te->owner);
WriteStr(AH, "false");
@@ -2778,6 +2782,9 @@ ReadToc(ArchiveHandle *AH)
if (AH->version >= K_VERS_1_16)
te->relkind = ReadInt(AH);
+ if (AH->version >= K_VERS_1_17)+ te->toasttype = ReadStr(AH);+
te->owner = ReadStr(AH);
is_supported = true;
if (AH->version < K_VERS_1_9)
@@ -3477,6 +3484,9 @@ _reconnectToDB(ArchiveHandle *AH, const char *dbname)
free(AH->currTablespace);
AH->currTablespace = NULL;
+ free(AH->currToastType);+ AH->currToastType = NULL;+
/* re-establish fixed state */
_doSetFixedOutputState(AH);
}
@@ -3682,6 +3692,56 @@ _selectTableAccessMethod(ArchiveHandle *AH, const char *tableam)
AH->currTableAm = pg_strdup(want);
}
++/*+ * Set the proper default_toast_type value for the table.+ */+static void+_selectToastType(ArchiveHandle *AH, const char *toasttype)+{+ RestoreOptions *ropt = AH->public.ropt;+ PQExpBuffer cmd;+ const char *want,+ *have;++ /* do nothing in --no-toast-type mode */+ if (ropt->noToastType)+ return;++ have = AH->currToastType;+ want = toasttype;++ if (!want)+ return;++ if (have && strcmp(want, have) == 0)+ return;++ cmd = createPQExpBuffer();++ appendPQExpBuffer(cmd, "SET default_toast_type = %s;", fmtId(toasttype));++ if (RestoringToDB(AH))+ {+ PGresult *res;++ res = PQexec(AH->connection, cmd->data);++ if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)+ warn_or_exit_horribly(AH,+ "could not set \"default_toast_type\": %s",+ PQerrorMessage(AH->connection));+ PQclear(res);+ }+ else+ ahprintf(AH, "%s\n\n", cmd->data);++ destroyPQExpBuffer(cmd);++ free(AH->currToastType);+ AH->currToastType = pg_strdup(want);+}+
/*
* Set the proper default table access method for a table without storage.
* Currently, this is required only for partitioned tables with a table AM.
@@ -3837,13 +3897,16 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx)
* Select owner, schema, tablespace and default AM as necessary. The
* default access method for partitioned tables is handled after
* generating the object definition, as it requires an ALTER command
- * rather than SET.+ * rather than SET. Partitioned tables do not have TOAST tables.
*/
_becomeOwner(AH, te);
_selectOutputSchema(AH, te->namespace);
_selectTablespace(AH, te->tablespace);
if (te->relkind != RELKIND_PARTITIONED_TABLE)
+ {
_selectTableAccessMethod(AH, te->tableam);
+ _selectToastType(AH, te->toasttype);+ }
/* Emit header comment for item */
if (!AH->noTocComments)
@@ -4402,6 +4465,8 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
AH->currTablespace = NULL;
free(AH->currTableAm);
AH->currTableAm = NULL;
+ free(AH->currToastType);+ AH->currToastType = NULL;
}
/*
@@ -5139,6 +5204,7 @@ CloneArchive(ArchiveHandle *AH)
clone->currSchema = NULL;
clone->currTableAm = NULL;
clone->currTablespace = NULL;
+ clone->currToastType = NULL;
/* savedPassword must be local in case we change it while connecting */
if (clone->savedPassword)
@@ -5198,6 +5264,7 @@ DeCloneArchive(ArchiveHandle *AH)
free(AH->currSchema);
free(AH->currTablespace);
free(AH->currTableAm);
+ free(AH->currToastType);
free(AH->savedPassword);
free(AH);
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.hindex 325b53fc9bd4..a9f8f75d4382 100644--- a/src/bin/pg_dump/pg_backup_archiver.h+++ b/src/bin/pg_dump/pg_backup_archiver.h@@ -71,10 +71,11 @@
#define K_VERS_1_16 MAKE_ARCHIVE_VERSION(1, 16, 0) /* BLOB METADATA entries
* and multiple BLOBS,
* relkind */
+#define K_VERS_1_17 MAKE_ARCHIVE_VERSION(1, 17, 0) /* TOAST type */
/* Current archive version number (the format we can output) */
#define K_VERS_MAJOR 1
-#define K_VERS_MINOR 16+#define K_VERS_MINOR 17
#define K_VERS_REV 0
#define K_VERS_SELF MAKE_ARCHIVE_VERSION(K_VERS_MAJOR, K_VERS_MINOR, K_VERS_REV)
@@ -325,6 +326,7 @@ struct _archiveHandle
char *currSchema; /* current schema, or NULL */
char *currTablespace; /* current tablespace, or NULL */
char *currTableAm; /* current table access method, or NULL */
+ char *currToastType; /* current TOAST type, or NULL */
/* in --transaction-size mode, this counts objects emitted in cur xact */
int txnCount;
@@ -359,6 +361,7 @@ struct _tocEntry
char *tablespace; /* null if not in a tablespace; empty string
* means use database default */
char *tableam; /* table access method, only for TABLE tags */
+ char *toasttype; /* TOAST table type, only for TABLE tags */
char relkind; /* relation kind, only for TABLE tags */
char *owner;
char *desc;
@@ -404,6 +407,7 @@ typedef struct _archiveOpts
const char *namespace;
const char *tablespace;
const char *tableam;
+ const char *toasttype;
char relkind;
const char *owner;
const char *description;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.cindex f3a353a61a58..7368cbb936b4 100644--- a/src/bin/pg_dump/pg_dump.c+++ b/src/bin/pg_dump/pg_dump.c@@ -506,6 +506,7 @@ main(int argc, char **argv)
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-table-access-method", no_argument, &dopt.outputNoTableAm, 1},
{"no-tablespaces", no_argument, &dopt.outputNoTablespaces, 1},
+ {"no-toast-type", no_argument, &dopt.outputNoToastType, 1},
{"quote-all-identifiers", no_argument, "e_all_identifiers, 1},
{"load-via-partition-root", no_argument, &dopt.load_via_partition_root, 1},
{"role", required_argument, NULL, 3},
@@ -1215,6 +1216,7 @@ main(int argc, char **argv)
ropt->noOwner = dopt.outputNoOwner;
ropt->noTableAm = dopt.outputNoTableAm;
ropt->noTablespace = dopt.outputNoTablespaces;
+ ropt->noToastType = dopt.outputNoToastType;
ropt->disable_triggers = dopt.disable_triggers;
ropt->use_setsessauth = dopt.use_setsessauth;
ropt->disable_dollar_quoting = dopt.disable_dollar_quoting;
@@ -1337,6 +1339,7 @@ help(const char *progname)
printf(_(" --no-table-access-method do not dump table access methods\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-toast-compression do not dump TOAST compression methods\n"));
+ printf(_(" --no-toast-type do not dump TOAST table type\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
@@ -7099,6 +7102,7 @@ getTables(Archive *fout, int *numTables)
int i_relfrozenxid;
int i_toastfrozenxid;
int i_toastoid;
+ int i_toasttype;
int i_relminmxid;
int i_toastminmxid;
int i_reloptions;
@@ -7153,6 +7157,14 @@ getTables(Archive *fout, int *numTables)
"ELSE 0 END AS foreignserver, "
"c.relfrozenxid, tc.relfrozenxid AS tfrozenxid, "
"tc.oid AS toid, "
+ "CASE WHEN c.reltoastrelid <> 0 THEN "+ " (SELECT CASE "+ " WHEN a.atttypid::regtype = 'oid'::regtype THEN 'oid'::text "+ " WHEN a.atttypid::regtype = 'bigint'::regtype THEN 'int8'::text "+ " ELSE NULL END"+ " FROM pg_attribute AS a "+ " WHERE a.attrelid = tc.oid AND a.attname = 'chunk_id') "+ " ELSE NULL END AS toasttype, "
"tc.relpages AS toastpages, "
"tc.reloptions AS toast_reloptions, "
"d.refobjid AS owning_tab, "
@@ -7323,6 +7335,7 @@ getTables(Archive *fout, int *numTables)
i_relfrozenxid = PQfnumber(res, "relfrozenxid");
i_toastfrozenxid = PQfnumber(res, "tfrozenxid");
i_toastoid = PQfnumber(res, "toid");
+ i_toasttype = PQfnumber(res, "toasttype");
i_relminmxid = PQfnumber(res, "relminmxid");
i_toastminmxid = PQfnumber(res, "tminmxid");
i_reloptions = PQfnumber(res, "reloptions");
@@ -7401,6 +7414,10 @@ getTables(Archive *fout, int *numTables)
tblinfo[i].frozenxid = atooid(PQgetvalue(res, i, i_relfrozenxid));
tblinfo[i].toast_frozenxid = atooid(PQgetvalue(res, i, i_toastfrozenxid));
tblinfo[i].toast_oid = atooid(PQgetvalue(res, i, i_toastoid));
+ if (PQgetisnull(res, i, i_toasttype))+ tblinfo[i].toast_type = NULL;+ else+ tblinfo[i].toast_type = pg_strdup(PQgetvalue(res, i, i_toasttype));
tblinfo[i].minmxid = atooid(PQgetvalue(res, i, i_relminmxid));
tblinfo[i].toast_minmxid = atooid(PQgetvalue(res, i, i_toastminmxid));
tblinfo[i].reloptions = pg_strdup(PQgetvalue(res, i, i_reloptions));
@@ -17862,6 +17879,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
{
char *tablespace = NULL;
char *tableam = NULL;
+ char *toasttype = NULL;
/*
* _selectTablespace() relies on tablespace-enabled objects in the
@@ -17876,12 +17894,15 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
if (RELKIND_HAS_TABLE_AM(tbinfo->relkind) ||
tbinfo->relkind == RELKIND_PARTITIONED_TABLE)
tableam = tbinfo->amname;
+ if (OidIsValid(tbinfo->toast_oid))+ toasttype = tbinfo->toast_type;
ArchiveEntry(fout, tbinfo->dobj.catId, tbinfo->dobj.dumpId,
ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
.namespace = tbinfo->dobj.namespace->dobj.name,
.tablespace = tablespace,
.tableam = tableam,
+ .toasttype = toasttype,
.relkind = tbinfo->relkind,
.owner = tbinfo->rolname,
.description = reltypename,
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.hindex dde85ed156cc..26ad5425bd4c 100644--- a/src/bin/pg_dump/pg_dump.h+++ b/src/bin/pg_dump/pg_dump.h@@ -325,6 +325,7 @@ typedef struct _tableInfo
uint32 frozenxid; /* table's relfrozenxid */
uint32 minmxid; /* table's relminmxid */
Oid toast_oid; /* toast table's OID, or 0 if none */
+ char *toast_type; /* toast table type, or NULL if none */
uint32 toast_frozenxid; /* toast table's relfrozenxid, if any */
uint32 toast_minmxid; /* toast table's relminmxid */
int ncheck; /* # of CHECK expressions */
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.cindex 27aa1b656989..20834c3094d3 100644--- a/src/bin/pg_dump/pg_dumpall.c+++ b/src/bin/pg_dump/pg_dumpall.c@@ -93,6 +93,7 @@ static int if_exists = 0;
static int inserts = 0;
static int no_table_access_method = 0;
static int no_tablespaces = 0;
+static int no_toast_type = 0;
static int use_setsessauth = 0;
static int no_comments = 0;
static int no_policies = 0;
@@ -162,6 +163,7 @@ main(int argc, char *argv[])
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-table-access-method", no_argument, &no_table_access_method, 1},
{"no-tablespaces", no_argument, &no_tablespaces, 1},
+ {"no-toast-type", no_argument, &no_tablespaces, 1},
{"quote-all-identifiers", no_argument, "e_all_identifiers, 1},
{"load-via-partition-root", no_argument, &load_via_partition_root, 1},
{"role", required_argument, NULL, 3},
@@ -445,6 +447,8 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " --no-table-access-method");
if (no_tablespaces)
appendPQExpBufferStr(pgdumpopts, " --no-tablespaces");
+ if (no_toast_type)+ appendPQExpBufferStr(pgdumpopts, " --no-toast-type");
if (quote_all_identifiers)
appendPQExpBufferStr(pgdumpopts, " --quote-all-identifiers");
if (load_via_partition_root)
@@ -699,6 +703,7 @@ help(void)
printf(_(" --no-table-access-method do not dump table access methods\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-toast-compression do not dump TOAST compression methods\n"));
+ printf(_(" --no-toast-type do not dump TOAST table types\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.cindex 6c129278bc52..df1c6e9eee88 100644--- a/src/bin/pg_dump/pg_restore.c+++ b/src/bin/pg_dump/pg_restore.c@@ -71,6 +71,7 @@ main(int argc, char **argv)
static int no_data_for_failed_tables = 0;
static int outputNoTableAm = 0;
static int outputNoTablespaces = 0;
+ static int outputNoToastType = 0;
static int use_setsessauth = 0;
static int no_comments = 0;
static int no_data = 0;
@@ -124,6 +125,7 @@ main(int argc, char **argv)
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
{"no-table-access-method", no_argument, &outputNoTableAm, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"no-toast-type", no_argument, &outputNoToastType, 1},
{"role", required_argument, NULL, 2},
{"section", required_argument, NULL, 3},
{"strict-names", no_argument, &strict_names, 1},
@@ -415,6 +417,7 @@ main(int argc, char **argv)
opts->noDataForFailedTables = no_data_for_failed_tables;
opts->noTableAm = outputNoTableAm;
opts->noTablespace = outputNoTablespaces;
+ opts->noToastType = outputNoToastType;
opts->use_setsessauth = use_setsessauth;
opts->no_comments = no_comments;
opts->no_policies = no_policies;
@@ -546,6 +549,7 @@ usage(const char *progname)
printf(_(" --no-subscriptions do not restore subscriptions\n"));
printf(_(" --no-table-access-method do not restore table access methods\n"));
printf(_(" --no-tablespaces do not restore tablespace assignments\n"));
+ printf(_(" --no-toast-type do not restore TOAST table types\n"));
printf(_(" --section=SECTION restore named section (pre-data, data, or post-data)\n"));
printf(_(" --statistics restore the statistics\n"));
printf(_(" --statistics-only restore only the statistics, not schema or data\n"));
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.plindex a86b38466de1..3c5d7c959356 100644--- a/src/bin/pg_dump/t/002_pg_dump.pl+++ b/src/bin/pg_dump/t/002_pg_dump.pl@@ -659,6 +659,15 @@ my %pgdump_runs = (
'postgres',
],
},
+ no_toast_type => {+ dump_cmd => [+ 'pg_dump', '--no-sync',+ '--file' => "$tempdir/no_toast_type.sql",+ '--no-toast-type',+ '--statistics',+ 'postgres',+ ],+ },
only_dump_test_schema => {
dump_cmd => [
'pg_dump', '--no-sync',
@@ -874,6 +883,7 @@ my %full_runs = (
no_privs => 1,
no_statistics => 1,
no_table_access_method => 1,
+ no_toast_type => 1,
pg_dumpall_dbprivs => 1,
pg_dumpall_exclude => 1,
schema_only => 1,
@@ -4934,6 +4944,31 @@ my %tests = (
},
},
+ # Test the case of multiple TOAST table types.+ 'CREATE TABLE regress_toast_type' => {+ create_order => 13,+ create_sql => '+ SET default_toast_type = int8;+ CREATE TABLE dump_test.regress_toast_type_int8 (col1 text);+ SET default_toast_type = oid;+ CREATE TABLE dump_test.regress_toast_type_oid (col1 text);+ RESET default_toast_type;',+ regexp => qr/^+ \QSET default_toast_type = int8;\E+ (\n(?!SET[^;]+;)[^\n]*)*+ \n\QCREATE TABLE dump_test.regress_toast_type_int8 (\E+ \n\s+\Qcol1 text\E+ \n\);/xm,+ like => {+ %full_runs, %dump_test_schema_runs, section_pre_data => 1,+ },+ unlike => {+ exclude_dump_test_schema => 1,+ no_toast_type => 1,+ only_dump_measurement => 1,+ },+ },+
#
# TABLE and MATVIEW stats will end up in SECTION_DATA.
# INDEX stats (expression columns only) will end up in SECTION_POST_DATA.
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgmlindex 0bc7609bdf81..c78d0b33c878 100644--- a/doc/src/sgml/ref/pg_dump.sgml+++ b/doc/src/sgml/ref/pg_dump.sgml@@ -1208,6 +1208,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>+ <term><option>--no-toast-type</option></term>+ <listitem>+ <para>+ Do not output commands to set <acronym>TOAST</acronym> table+ types.+ With this option, all <acronym>TOAST</acronym> tables will be+ restored with the default type.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry>
<term><option>--no-unlogged-table-data</option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgmlindex 364442f00f28..2ae4c503a8af 100644--- a/doc/src/sgml/ref/pg_dumpall.sgml+++ b/doc/src/sgml/ref/pg_dumpall.sgml@@ -550,6 +550,18 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>+ <term><option>--no-toast-type</option></term>+ <listitem>+ <para>+ Do not output commands to set <acronym>TOAST</acronym> table+ types.+ With this option, all <acronym>TOAST</acronym> tables will be+ restored with the default type.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry>
<term><option>--no-unlogged-table-data</option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgmlindex 261ead150395..c85e7e015df0 100644--- a/doc/src/sgml/ref/pg_restore.sgml+++ b/doc/src/sgml/ref/pg_restore.sgml@@ -796,6 +796,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>+ <term><option>--no-toast-type</option></term>+ <listitem>+ <para>+ Do not output commands to select <acronym>TOAST</acronym> table+ types.+ With this option, all <acronym>TOAST</acronym> tables will be+ created with whichever type is the default during restore.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry>
<term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term>
<listitem>
--
2.50.0
v4-0014-Add-new-vartag_external-for-8-byte-TOAST-values.patch (text/x-diff, 15.5K) download
From b1be9990e57bc2e2c33ad7feb076a6d0cf521625 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 Aug 2025 16:19:33 +0900
Subject: [PATCH v4 14/15] Add new vartag_external for 8-byte TOAST values
This is a new type of external TOAST pointer, able to be fed 8-byte
TOAST values. It uses a dedicated vartag_external, which is used when
a TOAST table uses bigint for its chunk_id.
The relevant callbacks are added to toast_external.c.
---
src/include/access/heaptoast.h | 8 +-
src/include/varatt.h | 34 +++-
src/backend/access/common/toast_external.c | 182 ++++++++++++++++--
src/backend/access/common/toast_internals.c | 1 -
src/backend/access/heap/heaptoast.c | 2 +
.../replication/logical/reorderbuffer.c | 10 +-
doc/src/sgml/storage.sgml | 6 +-
contrib/amcheck/verify_heapam.c | 2 +-
8 files changed, 225 insertions(+), 20 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.hindex 673e96f5488c..a39ad79a5ae9 100644--- a/src/include/access/heaptoast.h+++ b/src/include/access/heaptoast.h@@ -81,6 +81,12 @@
#define EXTERN_TUPLE_MAX_SIZE MaximumBytesPerTuple(EXTERN_TUPLES_PER_PAGE)
+#define TOAST_MAX_CHUNK_SIZE_INT8 \+ (EXTERN_TUPLE_MAX_SIZE - \+ MAXALIGN(SizeofHeapTupleHeader) - \+ (sizeof(uint32) * 2) - \+ sizeof(int32) - \+ VARHDRSZ)
#define TOAST_MAX_CHUNK_SIZE_OID \
(EXTERN_TUPLE_MAX_SIZE - \
MAXALIGN(SizeofHeapTupleHeader) - \
@@ -89,7 +95,7 @@
VARHDRSZ)
/* Maximum size of chunk possible */
-#define TOAST_MAX_CHUNK_SIZE TOAST_MAX_CHUNK_SIZE_OID+#define TOAST_MAX_CHUNK_SIZE Max(TOAST_MAX_CHUNK_SIZE_INT8, TOAST_MAX_CHUNK_SIZE_OID)
/* ----------
* heap_toast_insert_or_update -
diff --git a/src/include/varatt.h b/src/include/varatt.hindex 631aa2ecc494..a4b85031f06c 100644--- a/src/include/varatt.h+++ b/src/include/varatt.h@@ -41,6 +41,27 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*+ * struct varatt_external_int8 is a "larger" version of "TOAST pointer",+ * that uses an 8-byte integer as value.+ *+ * This follows the same properties as varatt_external_oid, except that+ * this is used in TOAST relations with int8 as attribute for chunk_id.+ */+typedef struct varatt_external_int8+{+ int32 va_rawsize; /* Original data size (includes header) */+ uint32 va_extinfo; /* External saved size (without header) and+ * compression method */+ /*+ * Unique ID of value within TOAST table, as two uint32 for alignment+ * and padding.+ */+ uint32 va_valueid_lo;+ uint32 va_valueid_hi;+ Oid va_toastrelid; /* RelID of TOAST table containing it */+} varatt_external_int8;+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -90,6 +111,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_INT8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -111,6 +133,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_INT8)+ return sizeof(varatt_external_int8);
else
{
Assert(false);
@@ -367,11 +391,19 @@ VARATT_IS_EXTERNAL_ONDISK_OID(const void *PTR)
return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
+/* Is varlena datum a pointer to on-disk toasted data with int8 value? */+static inline bool+VARATT_IS_EXTERNAL_ONDISK_INT8(const void *PTR)+{+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_INT8;+}+
/* Is varlena datum a pointer to on-disk toasted data? */
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL_ONDISK_OID(PTR);+ return VARATT_IS_EXTERNAL_ONDISK_OID(PTR) ||+ VARATT_IS_EXTERNAL_ONDISK_INT8(PTR);
}
/* Is varlena datum an indirect pointer? */
diff --git a/src/backend/access/common/toast_external.c b/src/backend/access/common/toast_external.cindex 5c36e5a11392..f02a707d7671 100644--- a/src/backend/access/common/toast_external.c+++ b/src/backend/access/common/toast_external.c@@ -14,9 +14,24 @@
#include "postgres.h"
#include "access/detoast.h"
+#include "access/genam.h"
#include "access/heaptoast.h"
+#include "access/toast_counter.h"
#include "access/toast_external.h"
+#include "access/toast_type.h"
#include "catalog/catalog.h"
+#include "miscadmin.h"+#include "utils/fmgroids.h"+#include "utils/snapmgr.h"+#include "utils/lsyscache.h"+++/* Callbacks for VARTAG_ONDISK_INT8 */+static void ondisk_int8_to_external_data(struct varlena *attr,+ toast_external_data *data);+static struct varlena *ondisk_int8_create_external_data(toast_external_data data);+static uint64 ondisk_int8_get_new_value(Relation toastrel, Oid indexid,+ AttrNumber attnum);
/* Callbacks for VARTAG_ONDISK_OID */
static void ondisk_oid_to_external_data(struct varlena *attr,
@@ -27,7 +42,7 @@ static uint64 ondisk_oid_get_new_value(Relation toastrel, Oid indexid,
/*
* Decompressed size of an on-disk varlena; but note argument is a struct
- * varatt_external_oid.+ * varatt_external_oid or varatt_external_int8.
*/
static inline Size
varatt_external_oid_get_extsize(struct varatt_external_oid toast_pointer)
@@ -35,9 +50,15 @@ varatt_external_oid_get_extsize(struct varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
}
+static inline Size+varatt_external_int8_get_extsize(struct varatt_external_int8 toast_pointer)+{+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;+}+
/*
* Compression method of an on-disk varlena; but note argument is a struct
- * varatt_external_oid.+ * varatt_external_oid or varatt_external_int8.
*/
static inline uint32
varatt_external_oid_get_compress_method(struct varatt_external_oid toast_pointer)
@@ -45,6 +66,12 @@ varatt_external_oid_get_compress_method(struct varatt_external_oid toast_pointer
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
+static inline uint32+varatt_external_int8_get_compress_method(struct varatt_external_int8 toast_pointer)+{+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;+}+
/*
* Testing whether an externally-stored TOAST value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -59,6 +86,19 @@ varatt_external_oid_is_compressed(struct varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+static inline bool+varatt_external_int8_is_compressed(struct varatt_external_int8 toast_pointer)+{+ return varatt_external_int8_get_extsize(toast_pointer) <+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);+}++/*+ * Size of an EXTERNAL datum that contains a standard TOAST pointer+ * (int8 value).+ */+#define TOAST_POINTER_INT8_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_int8))+
/*
* Size of an EXTERNAL datum that contains a standard TOAST pointer (OID
* value).
@@ -79,6 +119,13 @@ varatt_external_oid_is_compressed(struct varatt_external_oid toast_pointer)
* individual fields.
*/
static const toast_external_info toast_external_infos[TOAST_EXTERNAL_INFO_SIZE] = {
+ [VARTAG_ONDISK_INT8] = {+ .toast_pointer_size = TOAST_POINTER_INT8_SIZE,+ .maximum_chunk_size = TOAST_MAX_CHUNK_SIZE_INT8,+ .to_external_data = ondisk_int8_to_external_data,+ .create_external_data = ondisk_int8_create_external_data,+ .get_new_value = ondisk_int8_get_new_value,+ },
[VARTAG_ONDISK_OID] = {
.toast_pointer_size = TOAST_POINTER_OID_SIZE,
.maximum_chunk_size = TOAST_MAX_CHUNK_SIZE_OID,
@@ -117,22 +164,31 @@ toast_external_info_get_pointer_size(uint8 tag)
uint8
toast_external_assign_vartag(Oid toastrelid, uint64 value)
{
- /*- * If dealing with a code path where a TOAST relation may not be assigned,- * like heap_toast_insert_or_update(), just use the legacy- * vartag_external.- */- if (!OidIsValid(toastrelid))- return VARTAG_ONDISK_OID;+ Oid toast_typid;
/*
- * Currently there is only one type of vartag_external supported: 4-byte- * value with OID for the chunk_id type.+ * If dealing with a code path where a TOAST relation may not be assigned+ * like heap_toast_insert_or_update(), just use the vartag_external that+ * can be guessed based on the GUC default_toast_type.
*
- * Note: This routine will be extended to be able to use multiple- * vartag_external within a single TOAST relation type, that may change- * depending on the value used.+ * In bootstrap mode, we should not do any kind of syscache lookups,+ * so do the same and rely on the value of default_toast_type.
*/
+ if (!OidIsValid(toastrelid) || IsBootstrapProcessingMode())+ {+ if (default_toast_type == TOAST_TYPE_INT8)+ return VARTAG_ONDISK_INT8;+ return VARTAG_ONDISK_OID;+ }++ /*+ * Two types of vartag_external are currently supported: OID and int8,+ * which depend on the type assigned to "chunk_id" for the TOAST table.+ */+ toast_typid = get_atttype(toastrelid, 1);+ if (toast_typid == INT8OID)+ return VARTAG_ONDISK_INT8;+
return VARTAG_ONDISK_OID;
}
@@ -141,6 +197,104 @@ toast_external_assign_vartag(Oid toastrelid, uint64 value)
* the in-memory representation toast_external_data used in the backend.
*/
+/* Callbacks for VARTAG_ONDISK_INT8 */+static void+ondisk_int8_to_external_data(struct varlena *attr, toast_external_data *data)+{+ varatt_external_int8 external;++ VARATT_EXTERNAL_GET_POINTER(external, attr);+ data->rawsize = external.va_rawsize;++ /* External size and compression methods are stored in the same field */+ if (varatt_external_int8_is_compressed(external))+ {+ data->extsize = varatt_external_int8_get_extsize(external);+ data->compression_method = varatt_external_int8_get_compress_method(external);+ }+ else+ {+ data->extsize = external.va_extinfo;+ data->compression_method = TOAST_INVALID_COMPRESSION_ID;+ }++ data->value = (((uint64) external.va_valueid_hi) << 32) |+ external.va_valueid_lo;+ data->toastrelid = external.va_toastrelid;++}++static struct varlena *+ondisk_int8_create_external_data(toast_external_data data)+{+ struct varlena *result = NULL;+ varatt_external_int8 external;++ external.va_rawsize = data.rawsize;++ if (data.compression_method != TOAST_INVALID_COMPRESSION_ID)+ {+ /* Set size and compression method, in a single field. */+ VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(external,+ data.extsize,+ data.compression_method);+ }+ else+ external.va_extinfo = data.extsize;++ external.va_toastrelid = data.toastrelid;+ external.va_valueid_hi = (((uint64) data.value) >> 32);+ external.va_valueid_lo = (uint32) data.value;++ result = (struct varlena *) palloc(TOAST_POINTER_INT8_SIZE);+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_INT8);+ memcpy(VARDATA_EXTERNAL(result), &external, sizeof(external));++ return result;+}++static uint64+ondisk_int8_get_new_value(Relation toastrel, Oid indexid,+ AttrNumber attnum)+{+ uint64 new_value;+ SysScanDesc scan;+ ScanKeyData key;+ bool collides = false;++retry:+ new_value = GetNewToastId();++ /* No indexes in bootstrap mode, so leave */+ if (IsBootstrapProcessingMode())+ return new_value;++ Assert(IsSystemRelation(toastrel));++ CHECK_FOR_INTERRUPTS();++ /*+ * Check if the new value picked already exists in the toast relation.+ * If there is a conflict, retry.+ */+ ScanKeyInit(&key,+ attnum,+ BTEqualStrategyNumber, F_INT8EQ,+ Int64GetDatum(new_value));++ /* see notes in GetNewOidWithIndex() above about using SnapshotAny */+ scan = systable_beginscan(toastrel, indexid, true,+ SnapshotAny, 1, &key);+ collides = HeapTupleIsValid(systable_getnext(scan));+ systable_endscan(scan);++ if (collides)+ goto retry;++ return new_value;+}++
/* Callbacks for VARTAG_ONDISK_OID */
static void
ondisk_oid_to_external_data(struct varlena *attr, toast_external_data *data)
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex 98a240242127..ee7ef99181b3 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -18,7 +18,6 @@
#include "access/heapam.h"
#include "access/heaptoast.h"
#include "access/table.h"
-#include "access/toast_counter.h"
#include "access/toast_external.h"
#include "access/toast_internals.h"
#include "access/xact.h"
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex a01c1d627c6f..e3ace3acc9b9 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -31,7 +31,9 @@
#include "access/toast_external.h"
#include "access/toast_helper.h"
#include "access/toast_internals.h"
+#include "access/toast_type.h"
#include "utils/fmgroids.h"
+#include "utils/syscache.h"
/* ----------
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.cindex be33e7de6f8d..6c6526ab9c74 100644--- a/src/backend/replication/logical/reorderbuffer.c+++ b/src/backend/replication/logical/reorderbuffer.c@@ -4971,14 +4971,22 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
TupleDesc desc = RelationGetDescr(relation);
uint64 chunk_id;
int32 chunk_seq;
+ Oid toast_typid;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
+ toast_typid = TupleDescAttr(desc, 0)->atttypid;
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));+ /* This depends on the type of TOAST value dealt with. */+ if (toast_typid == OIDOID)+ chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));+ else if (toast_typid == INT8OID)+ chunk_id = DatumGetUInt64(fastgetattr(newtup, 1, desc, &isnull));+ else+ Assert(false);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 564783a1c559..29ba80e8423c 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -415,7 +415,11 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_MAX_CHUNK_SIZE_OID</symbol> bytes (by default this value is chosen+most <symbol>TOAST_MAX_CHUNK_SIZE_OID</symbol> bytes if the+<acronym>TOAST</acronym> relation uses the <literal>oid</literal> type for+<literal>chunk_id</literal>, or <symbol>TOAST_MAX_CHUNK_SIZE_INT8</symbol>+bytes if the <acronym>TOAST</acronym> relation uses the <literal>int8</literal>+type for <literal>chunk_id</literal> (by default these values are chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 833811c75437..958b1451b4ff 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -1733,7 +1733,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_INT8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
--
2.50.0
v4-0015-amcheck-Add-test-cases-for-8-byte-TOAST-values.patch (text/x-diff, 3.1K) download
From 93ee230abe55e39a811507847c292baed0e8fd2a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 19 Jun 2025 13:09:11 +0900
Subject: [PATCH v4 15/15] amcheck: Add test cases for 8-byte TOAST values
This patch is a proof of concept to show what is required to change in
the tests of pg_amcheck to be able to work with the new type of external
TOAST pointer.
---
src/bin/pg_amcheck/t/004_verify_heapam.pl | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/src/bin/pg_amcheck/t/004_verify_heapam.pl b/src/bin/pg_amcheck/t/004_verify_heapam.plindex 72693660fb64..5f82608b5c72 100644--- a/src/bin/pg_amcheck/t/004_verify_heapam.pl+++ b/src/bin/pg_amcheck/t/004_verify_heapam.pl@@ -83,8 +83,8 @@ use Test::More;
# it is convenient enough to do it this way. We define packing code
# constants here, where they can be compared easily against the layout.
-use constant HEAPTUPLE_PACK_CODE => 'LLLSSSSSCCLLCCCCCCCCCCllLL';-use constant HEAPTUPLE_PACK_LENGTH => 58; # Total size+use constant HEAPTUPLE_PACK_CODE => 'LLLSSSSSCCLLCCCCCCCCCCllLLL';+use constant HEAPTUPLE_PACK_LENGTH => 62; # Total size
# Read a tuple of our table from a heap page.
#
@@ -129,7 +129,8 @@ sub read_tuple
c_va_vartag => shift,
c_va_rawsize => shift,
c_va_extinfo => shift,
- c_va_valueid => shift,+ c_va_valueid_lo => shift,+ c_va_valueid_hi => shift,
c_va_toastrelid => shift);
# Stitch together the text for column 'b'
$tup{b} = join('', map { chr($tup{"b_body$_"}) } (1 .. 7));
@@ -163,7 +164,8 @@ sub write_tuple
$tup->{b_body6}, $tup->{b_body7},
$tup->{c_va_header}, $tup->{c_va_vartag},
$tup->{c_va_rawsize}, $tup->{c_va_extinfo},
- $tup->{c_va_valueid}, $tup->{c_va_toastrelid});+ $tup->{c_va_valueid_lo}, $tup->{c_va_valueid_hi},+ $tup->{c_va_toastrelid});
sysseek($fh, $offset, 0)
or BAIL_OUT("sysseek failed: $!");
defined(syswrite($fh, $buffer, HEAPTUPLE_PACK_LENGTH))
@@ -184,6 +186,7 @@ my $node = PostgreSQL::Test::Cluster->new('test');
$node->init(no_data_checksums => 1);
$node->append_conf('postgresql.conf', 'autovacuum=off');
$node->append_conf('postgresql.conf', 'max_prepared_transactions=10');
+$node->append_conf('postgresql.conf', 'default_toast_type=int8');
# Start the node and load the extensions. We depend on both
# amcheck and pageinspect for this test.
@@ -496,7 +499,7 @@ for (my $tupidx = 0; $tupidx < $ROWCOUNT; $tupidx++)
$tup->{t_hoff} += 128;
push @expected,
- qr/${$header}data begins at offset 152 beyond the tuple length 58/,+ qr/${$header}data begins at offset 152 beyond the tuple length 62/,
qr/${$header}tuple data should begin at byte 24, but actually begins at byte 152 \(3 attributes, no nulls\)/;
}
elsif ($offnum == 6)
@@ -581,7 +584,7 @@ for (my $tupidx = 0; $tupidx < $ROWCOUNT; $tupidx++)
elsif ($offnum == 13)
{
# Corrupt the bits in column 'c' toast pointer
- $tup->{c_va_valueid} = 0xFFFFFFFF;+ $tup->{c_va_valueid_lo} = 0xFFFFFFFF;
$header = header(0, $offnum, 2);
push @expected, qr/${header}toast value \d+ not found in toast table/;
--
2.50.0
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: michael@paquier.xyz, greg@burd.me, hukutoc@gmail.com, hannuk@google.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)
In-Reply-To: <aJWska1_ZiXUIETX@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox