pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedSupport for 8-byte TOAST values, round two
72+ messages / 10 participants
[nested] [flat]
Commits from this thread:
2026-09-23 74b89f8f96 doc: Mention oid8 TOAST values in the limits appendix
2026-09-19 a3f4fc5bac Add test for logical decoding with an oid8 TOAST table and OID > 2^32
2026-09-16 04c4c1c3a9 pg_upgrade: Run test suite with initial OID higher than UINT32_MAX
2026-09-16 87c6ef14f1 Tweak tests related to oid8 in amcheck and test_decoding
2026-09-15 0d34736e4c amcheck: Fix thinko with check of TOASTed attribute
2026-09-15 6e58d6356f Add support for external on-disk TOAST pointers as oid8
2026-09-14 4b4b5eae90 doc: Mention REPACK in description of reloption toast_value_type
2026-09-14 1cd8669179 Check that a promoted standby keeps its 8-byte OID counter
2026-09-11 f33844889b Add relation option toast_value_type
2026-09-11 1497ea77ed Add static assertion to track the no-padding rule of varatt_external_oid
2026-09-11 34fcd735f1 Add support for TOAST chunk_id type in binary upgrades
2026-09-10 cb29861646 Widen Object ID counter to 8 bytes
2026-09-08 1a531f787f pg_resetwal: Add test for -o with negative value
2026-09-06 c5806f2165 Rename varatt_external to varatt_external_oid
2026-09-03 2b3d11aaed Minimize footprint of TOAST_MAX_CHUNK_SIZE in heap and amcheck code
* Support for 8-byte TOAST values, round two
@ 2026-05-08 06:07 Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-05-08 06:07 UTC (permalink / raw)
To: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi all,
This is a follow-up of the previous thread about $subject, with a
reworked patch set for discussion in v20, as I care about the subject:
https://www.postgresql.org/message-id/aFOnKHG7Wn-Srnpv@paquier.xyz
The main feedback of the previous thread is that the previous
implementation with its callbacks for each vartag was not liked much,
and their were concerns with pointer redirections and performance.
This patch set uses what I am calling the "brutal" approach, relying
on a vartag_external of a varlena or the atttype of the TOAST relation
to decide which external toast pointer we should use. This uses no
function pointers, and patches the code to deal with Oid or Oid8 TOAST
values where it matters. So, this time, performance cannot really be
an issue.
The patch set is structured so as all the ground work happens first
(most of it comes from the previous patch set, reorganized a bit), and
the introduction of the varatt pieces are last, based on the following
rules:
- A table can use a 8-byte TOAST value with a new reloption, named
toast_value_type that can be set to "oid" (default) or "oid8",
creating a TOAST table with a value of the assigned type. This
includes support for dumps as well as binary upgrades, so as the
atttype of the chunk_id of the TOAST table is preserved. A table with
a TOAST type assigned cannot be changed to a different type through a
VACUUM FULL or a rewrite, as a matter of implementation simplicity.
- Renames and cleanup of various areas related to varatt_external,
renaming things to use OID.
- The 8-byte TOAST values rely on a Oid8, whose value is retrieved
from the control file extended by 4 bytes. The code supports
wraparound of values so as we don't assign anything between 0 and
FirstNormalObjectId for the lower bytes, same way as before.
The last patch introduces a new vartag_external and the new
varatt_external_oid8, with an Oid8 as value. Well, not exactly, the
patch uses two uint32 fields so as the structure is packed without
padding, as of:
typedef struct varatt_external_oid8
{
int32 va_rawsize;
uint32 va_extinfo;
uint32 va_valueid_lo;
uint32 va_valueid_hi;
Oid va_toastrelid;
} varatt_external_oid8;
Note that if applying all the patches except the last one, the code
would use an varatt_external_oid with an oid8 TOAST table. This
works, the split is to make reviews easier. An oid8 TOAST table
always uses a varatt_external_oid8.
I have done a lot of back-and-forth in the patch to try to find a good
balance between the manipulation of the varlenas in the detoast and
compression paths, as well as reorderbuffer.c and amcheck. And I have
finished with the attached, which is kind of nice. The last patch has
a low footprint:
9 files changed, 537 insertions(+), 208 deletions(-)
Happy to discuss about all that at pgconf.dev. This is of course
intended for v20, added to the next CF.
Thanks,
--
Michael
Attachments: download all (12 files, .tar.gz)
v1-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-ins.patch (text/plain, 15.3K) download
From 3231b5914fca10587993625666421d6d9dfacbec Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 17:26:36 +0900
Subject: [PATCH v1 01/11] Refactor some TOAST value ID code to use Oid8
instead of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 14 +++--
contrib/amcheck/verify_heapam.c | 56 +++++++++++--------
6 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..ff234c652230 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..ea4a473ff03c 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1983,7 +1983,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 77d42e7ed65a..11e3d39d1bd4 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);
-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)
+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)
+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..747b8c56fc95 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -725,7 +725,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",
+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -737,13 +737,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -752,7 +752,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",
+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -781,7 +781,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",
+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 682d13c9f22f..252746022bf8 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */
+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -4976,7 +4976,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5000,7 +5000,7 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;
+ Oid8 chunk_id;
int32 chunk_seq;
if (txn->toast_hash == NULL)
@@ -5027,11 +5027,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5140,6 +5140,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5160,13 +5161,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,
+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 20ff58aa7825..b67fe0cc3f37 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1563,6 +1563,9 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
bool isnull;
int32 chunksize;
int32 expected_size;
+ Oid8 toast_valueid;
+
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1570,16 +1573,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",
+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",
+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1590,8 +1593,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",
+ toast_valueid,
chunk_seq));
return;
}
@@ -1610,8 +1613,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",
+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1622,8 +1625,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",
+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1633,8 +1636,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",
+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1665,6 +1668,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external toast_pointer;
@@ -1773,12 +1777,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",
- toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1805,16 +1810,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",
- toast_pointer.va_valueid, cmid));
+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",
+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",
+ toast_pointer_valueid));
return true;
}
@@ -1822,8 +1827,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",
+ toast_pointer_valueid));
return true;
}
@@ -1868,6 +1873,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;
@@ -1898,14 +1904,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;
+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " not found in toast table",
+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",
+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.54.0
v1-0002-Minimize-footprint-of-TOAST_MAX_CHUNK_SIZE-in-hea.patch (text/plain, 4.6K) download
From 770ddfad484ffcc555c0797465299677350a5c25 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 17:40:13 +0900
Subject: [PATCH v1 02/11] Minimize footprint of TOAST_MAX_CHUNK_SIZE in heap
and amcheck
This eases a follow-up change to support 8-byte TOAST value IDs, as the
maximum chunk size allowed for a single chunk of TOASTed data depends on
the size of the value ID.
---
src/backend/access/heap/heaptoast.c | 20 ++++++++++++--------
contrib/amcheck/verify_heapam.c | 13 +++++++++----
2 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 747b8c56fc95..d1f9c25e93cd 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -634,11 +634,12 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
SysScanDesc toastscan;
HeapTuple ttup;
int32 expectedchunk;
- int32 totalchunks = ((attrsize - 1) / TOAST_MAX_CHUNK_SIZE) + 1;
+ int32 totalchunks;
int startchunk;
int endchunk;
int num_indexes;
int validIndex;
+ int32 max_chunk_size;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -646,8 +647,11 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
- startchunk = sliceoffset / TOAST_MAX_CHUNK_SIZE;
- endchunk = (sliceoffset + slicelength - 1) / TOAST_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+
+ totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
+ startchunk = sliceoffset / max_chunk_size;
+ endchunk = (sliceoffset + slicelength - 1) / max_chunk_size;
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
@@ -747,8 +751,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
- expected_size = curchunk < totalchunks - 1 ? TOAST_MAX_CHUNK_SIZE
- : attrsize - ((totalchunks - 1) * TOAST_MAX_CHUNK_SIZE);
+ expected_size = curchunk < totalchunks - 1 ? max_chunk_size
+ : attrsize - ((totalchunks - 1) * max_chunk_size);
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -763,12 +767,12 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
chcpystrt = 0;
chcpyend = chunksize - 1;
if (curchunk == startchunk)
- chcpystrt = sliceoffset % TOAST_MAX_CHUNK_SIZE;
+ chcpystrt = sliceoffset % max_chunk_size;
if (curchunk == endchunk)
- chcpyend = (sliceoffset + slicelength - 1) % TOAST_MAX_CHUNK_SIZE;
+ chcpyend = (sliceoffset + slicelength - 1) % max_chunk_size;
memcpy(VARDATA(result) +
- curchunk * TOAST_MAX_CHUNK_SIZE - sliceoffset + chcpystrt,
+ (curchunk * max_chunk_size - sliceoffset) + chcpystrt,
chunkdata + chcpystrt,
(chcpyend - chcpystrt) + 1);
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index b67fe0cc3f37..22bba6161de4 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1558,15 +1558,19 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 extsize)
{
int32 chunk_seq;
- int32 last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;
+ int32 last_chunk_seq;
Pointer chunk;
bool isnull;
int32 chunksize;
int32 expected_size;
Oid8 toast_valueid;
+ int32 max_chunk_size;
toast_valueid = ta->toast_pointer.va_valueid;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
+
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
ctx->toast_rel->rd_att, &isnull));
@@ -1631,8 +1635,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
return;
}
- expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
- : extsize - (last_chunk_seq * TOAST_MAX_CHUNK_SIZE);
+ expected_size = chunk_seq < last_chunk_seq ? max_chunk_size
+ : extsize - (last_chunk_seq * max_chunk_size);
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
@@ -1874,9 +1878,10 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
Oid8 toast_valueid;
+ int32 max_chunk_size = TOAST_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
--
2.54.0
v1-0003-Rename-varatt_external-to-varatt_external_oid.patch (text/plain, 24.6K) download
From 17b2aecb080d74ec41519214bb44fc6214065cc8 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 18:28:10 +0900
Subject: [PATCH v1 03/11] Rename varatt_external to varatt_external_oid
This impacts a few more things, related to this structure:
- VARTAG_ONDISK -> VARTAG_ONDISK_OID
- TOAST_POINTER_SIZE -> TOAST_OID_POINTER_SIZE
- TOAST_MAX_CHUNK_SIZE -> TOAST_OID_MAX_CHUNK_SIZE
- Macros in varatt.h for varatt_external_oid.
This rename is in preparation of a follow-up commit that aims at adding
support for multiple types of external on-disk TOAST pointers, where the
OID type is only one subset of them.
---
src/include/access/detoast.h | 4 +--
src/include/access/heaptoast.h | 6 ++--
src/include/varatt.h | 36 ++++++++++---------
src/backend/access/common/detoast.c | 30 ++++++++--------
src/backend/access/common/toast_compression.c | 6 ++--
src/backend/access/common/toast_internals.c | 16 ++++-----
src/backend/access/heap/heaptoast.c | 2 +-
src/backend/access/table/toast_helper.c | 4 +--
src/backend/access/transam/xlog.c | 8 ++---
.../replication/logical/reorderbuffer.c | 6 ++--
src/backend/utils/adt/varlena.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 2 +-
doc/src/sgml/func/func-info.sgml | 2 +-
doc/src/sgml/storage.sgml | 2 +-
contrib/amcheck/verify_heapam.c | 14 ++++----
15 files changed, 72 insertions(+), 68 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..2b1428b1e81c 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -14,7 +14,7 @@
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
- * into a local "varatt_external" toast pointer. This should be
+ * into a local "varatt_external_oid" toast pointer. This should be
* just a memcpy, but some versions of gcc seem to produce broken code
* that assumes the datum contents are aligned. Introducing an explicit
* intermediate "varattrib_1b_e *" variable seems to fix it.
@@ -28,7 +28,7 @@ do { \
} while (0)
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
-#define TOAST_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external))
+#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index ff234c652230..aaae6e0fef69 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,19 +69,19 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_MAX_CHUNK_SIZE data bytes. This number *must*
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
* be small enough that the completed toast-table tuple (including the
* ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
#define EXTERN_TUPLE_MAX_SIZE MaximumBytesPerTuple(EXTERN_TUPLES_PER_PAGE)
-#define TOAST_MAX_CHUNK_SIZE \
+#define TOAST_OID_MAX_CHUNK_SIZE \
(EXTERN_TUPLE_MAX_SIZE - \
MAXALIGN(SizeofHeapTupleHeader) - \
sizeof(Oid) - \
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 000bdf33b923..33979bbaaad1 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -16,7 +16,7 @@
#define VARATT_H
/*
- * varatt_external is a traditional "TOAST pointer", that is, the
+ * varatt_external_oid is a traditional "TOAST pointer", that is, the
* information needed to fetch a Datum stored out-of-line in a TOAST table.
* The data is compressed if and only if the external size stored in
* va_extinfo is less than va_rawsize - VARHDRSZ.
@@ -29,14 +29,14 @@
* you can look at these fields! (The reason we use memcmp is to avoid
* having to do that just to detect equality of two TOAST pointers...)
*/
-typedef struct varatt_external
+typedef struct varatt_external_oid
{
int32 va_rawsize; /* Original data size (includes header) */
uint32 va_extinfo; /* External saved size (without header) and
* compression method */
Oid va_valueid; /* Unique ID of value within TOAST table */
Oid va_toastrelid; /* RelID of TOAST table containing it */
-} varatt_external;
+} varatt_external_oid;
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
@@ -51,7 +51,7 @@ typedef struct varatt_external
* The creator of such a Datum is entirely responsible that the referenced
* storage survives for as long as referencing pointer Datums can exist.
*
- * Note that just as for varatt_external, this struct is stored
+ * Note that just as for varatt_external_oid, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct varatt_indirect
@@ -66,7 +66,7 @@ typedef struct varatt_indirect
* storage. APIs for this, in particular the definition of struct
* ExpandedObjectHeader, are in src/include/utils/expandeddatum.h.
*
- * Note that just as for varatt_external, this struct is stored
+ * Note that just as for varatt_external_oid, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct ExpandedObjectHeader ExpandedObjectHeader;
@@ -78,15 +78,16 @@ typedef struct varatt_expanded
/*
* Type tag for the various sorts of "TOAST pointer" datums. The peculiar
- * value for VARTAG_ONDISK comes from a requirement for on-disk compatibility
- * with a previous notion that the tag field was the pointer datum's length.
+ * value for VARTAG_ONDISK_OID comes from a requirement for on-disk
+ * compatibility with a previous notion that the tag field was the pointer
+ * datum's length.
*/
typedef enum vartag_external
{
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
- VARTAG_ONDISK = 18
+ VARTAG_ONDISK_OID = 18
} vartag_external;
/* Is a TOAST pointer either type of expanded-object pointer? */
@@ -105,8 +106,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_indirect);
else if (VARTAG_IS_EXPANDED(tag))
return sizeof(varatt_expanded);
- else if (tag == VARTAG_ONDISK)
- return sizeof(varatt_external);
+ else if (tag == VARTAG_ONDISK_OID)
+ return sizeof(varatt_external_oid);
else
{
Assert(false);
@@ -360,7 +361,7 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK;
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
/* Is varlena datum an indirect pointer? */
@@ -502,15 +503,18 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
return ((const varattrib_4b *) PTR)->va_compressed.va_tcinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Same for external Datums; but note argument is a varatt_external */
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid.
+ */
static inline Size
-VARATT_EXTERNAL_GET_EXTSIZE(varatt_external toast_pointer)
+VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer)
{
return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
}
static inline uint32
-VARATT_EXTERNAL_GET_COMPRESS_METHOD(varatt_external toast_pointer)
+VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
{
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
@@ -533,9 +537,9 @@ VARATT_EXTERNAL_GET_COMPRESS_METHOD(varatt_external toast_pointer)
* actually saves space, so we expect either equality or less-than.
*/
static inline bool
-VARATT_EXTERNAL_IS_COMPRESSED(varatt_external toast_pointer)
+VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
{
- return VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) <
+ return VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer) <
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..a50aea7b6b12 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,12 +225,12 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +240,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,7 +251,7 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_GET_COMPRESS_METHOD(toast_pointer) ==
+ if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
@@ -344,7 +344,7 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
int32 attrsize;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
@@ -353,11 +353,11 @@ toast_fetch_datum(varlena *attr)
/* Must copy to access aligned fields */
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- attrsize = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
else
SET_VARSIZE(result, attrsize + VARHDRSZ);
@@ -398,7 +398,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
int32 attrsize;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
@@ -412,9 +412,9 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
+ Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
- attrsize = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
if (sliceoffset >= attrsize)
{
@@ -427,7 +427,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +440,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -550,7 +550,7 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
result = toast_pointer.va_rawsize;
@@ -610,10 +610,10 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 5a5d579494a2..3b1bd6519261 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -262,12 +262,12 @@ toast_get_compression_id(varlena *attr)
*/
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
- cmid = VARATT_EXTERNAL_GET_COMPRESS_METHOD(toast_pointer);
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ cmid = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
}
else if (VARATT_IS_COMPRESSED(attr))
cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 11e3d39d1bd4..ca5ed8dfabca 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -124,7 +124,7 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -176,7 +176,7 @@ toast_save_datum(Relation rel, Datum value,
VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer));
+ Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
}
else
{
@@ -225,7 +225,7 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_valueid = InvalidOid;
if (oldexternal != NULL)
{
- varatt_external old_toast_pointer;
+ varatt_external_oid old_toast_pointer;
Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
/* Must copy to access aligned fields */
@@ -289,7 +289,7 @@ toast_save_datum(Relation rel, Datum value,
{
alignas(int32) varlena hdr;
/* this is to make the union big enough for a chunk: */
- char data[TOAST_MAX_CHUNK_SIZE + VARHDRSZ];
+ char data[TOAST_OID_MAX_CHUNK_SIZE + VARHDRSZ];
} chunk_data;
int32 chunk_size;
@@ -298,7 +298,7 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
/*
* Build a tuple and store it
@@ -359,8 +359,8 @@ toast_save_datum(Relation rel, Datum value,
/*
* Create the TOAST pointer value that we'll return
*/
- result = (varlena *) palloc(TOAST_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK);
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
return PointerGetDatum(result);
@@ -376,7 +376,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index d1f9c25e93cd..81154c17376c 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -647,7 +647,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
- max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..6f2d691cc681 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -171,7 +171,7 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_POINTER_SIZE);
+ * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
* if not, no benefit is to be expected by compressing it.
*
* The return value is the index of the biggest suitable column, or
@@ -184,7 +184,7 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_POINTER_SIZE);
+ int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f0434da40c94..98c83be84ec9 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4316,7 +4316,7 @@ WriteControlFile(void)
ControlFile->nameDataLen = NAMEDATALEN;
ControlFile->indexMaxKeys = INDEX_MAX_KEYS;
- ControlFile->toast_max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ ControlFile->toast_max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
ControlFile->loblksize = LOBLKSIZE;
ControlFile->float8ByVal = true; /* vestigial */
@@ -4569,15 +4569,15 @@ ReadControlFile(void)
"INDEX_MAX_KEYS", ControlFile->indexMaxKeys,
"INDEX_MAX_KEYS", INDEX_MAX_KEYS),
errhint("It looks like you need to recompile or initdb.")));
- if (ControlFile->toast_max_chunk_size != TOAST_MAX_CHUNK_SIZE)
+ if (ControlFile->toast_max_chunk_size != TOAST_OID_MAX_CHUNK_SIZE)
ereport(FATAL,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("database files are incompatible with server"),
/* translator: %s is a variable name and %d is its value */
errdetail("The database cluster was initialized with %s %d,"
" but the server was compiled with %s %d.",
- "TOAST_MAX_CHUNK_SIZE", ControlFile->toast_max_chunk_size,
- "TOAST_MAX_CHUNK_SIZE", (int) TOAST_MAX_CHUNK_SIZE),
+ "TOAST_OID_MAX_CHUNK_SIZE", ControlFile->toast_max_chunk_size,
+ "TOAST_OID_MAX_CHUNK_SIZE", (int) TOAST_OID_MAX_CHUNK_SIZE),
errhint("It looks like you need to recompile or initdb.")));
if (ControlFile->loblksize != LOBLKSIZE)
ereport(FATAL,
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 252746022bf8..a7b524906995 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5134,7 +5134,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *varlena_pointer;
/* va_rawsize is the size of the original datum -- including header */
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
@@ -5204,10 +5204,10 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer));
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
/* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
else
SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index c0ff51bd2fc1..ae406f458a99 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4258,7 +4258,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index f8d25afed9de..571805cbd54c 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -733,7 +733,7 @@ GuessControlValues(void)
ControlFile.xlog_seg_size = DEFAULT_XLOG_SEG_SIZE;
ControlFile.nameDataLen = NAMEDATALEN;
ControlFile.indexMaxKeys = INDEX_MAX_KEYS;
- ControlFile.toast_max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ ControlFile.toast_max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
ControlFile.loblksize = LOBLKSIZE;
ControlFile.float8ByVal = true; /* vestigial */
diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml
index 00f64f50ceb8..3c6f2edd5e50 100644
--- a/doc/src/sgml/func/func-info.sgml
+++ b/doc/src/sgml/func/func-info.sgml
@@ -3596,7 +3596,7 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
</row>
<row>
- <entry><structfield>max_toast_chunk_size</structfield></entry>
+ <entry><structfield>max_toast_oid_chunk_size</structfield></entry>
<entry><type>integer</type></entry>
</row>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 6b6377503bf6..6b1e00bfd38e 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,7 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 22bba6161de4..119a54a9404c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,7 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1568,7 +1568,7 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
toast_valueid = ta->toast_pointer.va_valueid;
- max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
/* Sanity-check the sequence number. */
@@ -1674,7 +1674,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
uint16 infomask;
Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1733,7 +1733,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK)
+ if (va_tag != VARTAG_ONDISK_OID)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1791,7 +1791,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
{
ToastCompressionId cmid;
bool valid = false;
@@ -1878,9 +1878,9 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
Oid8 toast_valueid;
- int32 max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
- extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
--
2.54.0
v1-0004-Switch-pg_column_toast_chunk_id-return-value-from.patch (text/plain, 5.7K) download
From 68932b003d2e49a00d794d651ae0bfc5b0888756 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v1 04/11] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fa9ae79082b8..ce49fe25d46e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7856,7 +7856,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',
+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index ae406f458a99..0a2377667655 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4287,7 +4287,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);
+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out
index cb14ddcee34b..21859fc0dce2 100644
--- a/src/test/isolation/expected/cluster-toast-value-reuse.out
+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out
@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec
index 9a2d10600b39..96cf899db296 100644
--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec
+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec
@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
index fa76bcf28ef9..cc8f034d0167 100644
--- a/src/test/modules/injection_points/specs/repack_toast.spec
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);
+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index c3261bff209f..3d5ea1c256be 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 946ee5726cdd..7ce6ad62f090 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 72038fc835f4..1cbc97917d24 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>
+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.54.0
v1-0005-Add-catcache-support-for-OID8OID.patch (text/plain, 1.3K) download
From 881df941bdebe7287e7132e53408a689f9e895d2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 20:00:30 +0900
Subject: [PATCH v1 05/11] Add catcache support for OID8OID
This is required to be able to do catalog cache lookups of oid8 fields
for toast values of the same type.
---
src/backend/utils/cache/catcache.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index a8e7bf649d23..9afd8dee5c1a 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -248,6 +248,18 @@ int4hashfast(Datum datum)
return murmurhash32((int32) DatumGetInt32(datum));
}
+static bool
+oid8eqfast(Datum a, Datum b)
+{
+ return DatumGetObjectId8(a) == DatumGetObjectId8(b);
+}
+
+static uint32
+oid8hashfast(Datum datum)
+{
+ return murmurhash64(DatumGetObjectId8(datum));
+}
+
static bool
texteqfast(Datum a, Datum b)
{
@@ -311,6 +323,11 @@ GetCCHashEqFuncs(Oid keytype, CCHashFN *hashfunc, RegProcedure *eqfunc, CCFastEq
*fasteqfunc = int4eqfast;
*eqfunc = F_INT4EQ;
break;
+ case OID8OID:
+ *hashfunc = oid8hashfast;
+ *fasteqfunc = oid8eqfast;
+ *eqfunc = F_OID8EQ;
+ break;
case TEXTOID:
*hashfunc = texthashfast;
*fasteqfunc = texteqfast;
--
2.54.0
v1-0006-Add-support-for-TOAST-chunk_id-type-in-binary-upg.patch (text/plain, 9.3K) download
From 0f955879bf80f6274d0770466e25f87361a46d6b Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 10:57:59 +0900
Subject: [PATCH v1 06/11] Add support for TOAST chunk_id type in binary
upgrades
This commit adds a new function, which would set the type of a chunk_id
attribute for a TOAST table across upgrades. This piece currently works
only with chunk_id = OIDOID, but it is required in a follow-up patch
where support for chunk_id = OID8OID is supported on top of the existing
one.
---
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 ++++
src/backend/catalog/heap.c | 1 +
src/backend/catalog/toasting.c | 20 ++++++++++++++++++-
src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++++
src/bin/pg_dump/pg_dump.c | 10 +++++++++-
.../expected/spgist_name_ops.out | 6 ++++--
7 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae443859..117f3e3f7746 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -29,6 +29,7 @@ extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_index_pg_class_relfilenumber;
extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber;
+extern PGDLLIMPORT Oid binary_upgrade_next_toast_chunk_id_typoid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ce49fe25d46e..14ca38e559ea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11969,6 +11969,10 @@
proname => 'binary_upgrade_set_next_toast_pg_class_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_toast_pg_class_oid' },
+{ oid => '8219', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_toast_chunk_id_typoid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_toast_chunk_id_typoid' },
{ oid => '3589', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_set_next_pg_enum_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 7678ab13f6a8..866e4cfa835d 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -80,6 +80,7 @@
/* Potentially set by pg_upgrade_support functions */
Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid;
Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid;
+Oid binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
RelFileNumber binary_upgrade_next_heap_pg_class_relfilenumber = InvalidRelFileNumber;
RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber = InvalidRelFileNumber;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 4aa52a4bd253..bc0a72268335 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -146,6 +146,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_chunkid_typid = OIDOID;
/*
* Is it already toasted?
@@ -184,6 +185,23 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!OidIsValid(binary_upgrade_next_toast_pg_class_oid))
return false;
+
+ /*
+ * The attribute type for chunk_id should have been set when requesting
+ * a TOAST table creation.
+ */
+ if (!OidIsValid(binary_upgrade_next_toast_chunk_id_typoid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("toast chunk_id type not set while in binary upgrade mode")));
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
+ binary_upgrade_next_toast_chunk_id_typoid)));
+
+ toast_chunkid_typid = binary_upgrade_next_toast_chunk_id_typoid;
+ binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
}
/*
@@ -205,7 +223,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,
+ toast_chunkid_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4feeb..06b789d71ded 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -149,6 +149,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_toast_chunk_id_typoid(PG_FUNCTION_ARGS)
+{
+ Oid typoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_toast_chunk_id_typoid = typoid;
+
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d56dcc701ce8..82c2646e793b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -104,6 +104,7 @@ typedef struct
RelFileNumber relfilenumber; /* object filenode */
Oid toast_oid; /* toast table OID */
RelFileNumber toast_relfilenumber; /* toast table filenode */
+ Oid toast_chunk_id_typoid; /* type of chunk_id attribute */
Oid toast_index_oid; /* toast table index OID */
RelFileNumber toast_index_relfilenumber; /* toast table index filenode */
} BinaryUpgradeClassOidItem;
@@ -5918,7 +5919,10 @@ collectBinaryUpgradeClassOids(Archive *fout)
const char *query;
query = "SELECT c.oid, c.relkind, c.relfilenode, c.reltoastrelid, "
- "ct.relfilenode, i.indexrelid, cti.relfilenode "
+ "ct.relfilenode, i.indexrelid, cti.relfilenode, "
+ "(SELECT a.atttypid FROM pg_attribute AS a "
+ " WHERE a.attrelid = c.reltoastrelid AND attname = 'chunk_id'::text) "
+ " AS toastchunktypid "
"FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_index i "
"ON (c.reltoastrelid = i.indrelid AND i.indisvalid) "
"LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) "
@@ -5940,6 +5944,7 @@ collectBinaryUpgradeClassOids(Archive *fout)
binaryUpgradeClassOids[i].toast_relfilenumber = atooid(PQgetvalue(res, i, 4));
binaryUpgradeClassOids[i].toast_index_oid = atooid(PQgetvalue(res, i, 5));
binaryUpgradeClassOids[i].toast_index_relfilenumber = atooid(PQgetvalue(res, i, 6));
+ binaryUpgradeClassOids[i].toast_chunk_id_typoid = atooid(PQgetvalue(res, i, 7));
}
PQclear(res);
@@ -6004,6 +6009,9 @@ binary_upgrade_set_pg_class_oids(Archive *fout,
appendPQExpBuffer(upgrade_buffer,
"SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n",
entry->toast_relfilenumber);
+ appendPQExpBuffer(upgrade_buffer,
+ "SELECT pg_catalog.binary_upgrade_set_next_toast_chunk_id_typoid('%u'::pg_catalog.oid);\n",
+ entry->toast_chunk_id_typoid);
/* every toast table has an index */
appendPQExpBuffer(upgrade_buffer,
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede2430..35e59d0cd83c 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -61,9 +61,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -110,9 +111,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.54.0
v1-0007-Enlarge-OID-generation-to-8-bytes.patch (text/plain, 14.5K) download
From 5dbff74dd7b253261aafe3f03f07e2e3400277c4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:15:15 +0900
Subject: [PATCH v1 07/11] Enlarge OID generation to 8 bytes
This adds a new routine called GetNewObjectId8() in varsup.c, which is
able to retrieve a 64b OID. GetNewObjectId() is kept compatible with
its origin, where we still check that the lower 32 bits of the counter
do not wraparound, handling the FirstNormalObjectId case.
pg_resetwal -o/--next-oid is updated to be able to handle 8-byte OIDs.
---
src/include/access/transam.h | 3 +-
src/include/access/xlog.h | 2 +-
src/include/catalog/pg_control.h | 2 +-
src/backend/access/rmgrdesc/xlogdesc.c | 8 +--
src/backend/access/transam/varsup.c | 62 ++++++++++++++++-------
src/backend/access/transam/xlog.c | 8 +--
src/backend/access/transam/xlogrecovery.c | 2 +-
src/bin/pg_controldata/pg_controldata.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 10 ++--
doc/src/sgml/ref/pg_resetwal.sgml | 6 +--
10 files changed, 66 insertions(+), 39 deletions(-)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 55a4ab26b348..f687df1edc60 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -211,7 +211,7 @@ typedef struct TransamVariablesData
/*
* These fields are protected by OidGenLock.
*/
- Oid nextOid; /* next OID to assign */
+ Oid8 nextOid; /* next OID (8 bytes) to assign */
uint32 oidCount; /* OIDs available before must do XLOG work */
/*
@@ -353,6 +353,7 @@ extern void SetTransactionIdLimit(TransactionId oldest_datfrozenxid,
extern void AdvanceOldestClogXid(TransactionId oldest_datfrozenxid);
extern bool ForceTransactionIdLimitUpdate(void);
extern Oid GetNewObjectId(void);
+extern Oid8 GetNewObjectId8(void);
extern void StopGeneratingPinnedObjectIds(void);
#ifdef USE_ASSERT_CHECKING
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 4dd986242046..64763ffffc92 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -270,7 +270,7 @@ extern void ShutdownXLOG(int code, Datum arg);
extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
-extern void XLogPutNextOid(Oid nextOid);
+extern void XLogPutNextOid(Oid8 nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern XLogRecPtr XLogAssignLSN(void);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 80b3a730e035..7f708d53ea94 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -43,7 +43,7 @@ typedef struct CheckPoint
int wal_level; /* current wal_level */
bool logicalDecodingEnabled; /* current logical decoding status */
FullTransactionId nextXid; /* next free transaction ID */
- Oid nextOid; /* next free OID */
+ Oid8 nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 2468a7d25785..64e749c1e6b7 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -101,7 +101,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
CheckPoint *checkpoint = (CheckPoint *) rec;
appendStringInfo(buf, "redo %X/%08X; "
- "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "
+ "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid " OID8_FORMAT "; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
@@ -129,10 +129,10 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
else if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
- memcpy(&nextOid, rec, sizeof(Oid));
- appendStringInfo(buf, "%u", nextOid);
+ memcpy(&nextOid, rec, sizeof(Oid8));
+ appendStringInfo(buf, OID8_FORMAT, nextOid);
}
else if (info == XLOG_RESTORE_POINT)
{
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index dc5e32d86f34..b143d1918120 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -541,31 +541,51 @@ ForceTransactionIdLimitUpdate(void)
/*
- * GetNewObjectId -- allocate a new OID
+ * GetNewObjectId -- allocate a new OID (4 bytes)
*
- * OIDs are generated by a cluster-wide counter. Since they are only 32 bits
- * wide, counter wraparound will occur eventually, and therefore it is unwise
- * to assume they are unique unless precautions are taken to make them so.
- * Hence, this routine should generally not be used directly. The only direct
- * callers should be GetNewOidWithIndex() and GetNewRelFileNumber() in
- * catalog/catalog.c.
+ * OIDs are generated by a cluster-wide counter. The callers of this routine
+ * expect a 32 bit-wide counter, and counter wraparound will occur eventually,
+ * and therefore it is unwise to assume they are unique unless precautions are
+ * taken to make them so. This routine should generally not be used directly.
+ * The only direct callers should be GetNewOidWithIndex() and
+ * GetNewRelFileNumber() in catalog/catalog.c.
*/
Oid
GetNewObjectId(void)
{
- Oid result;
+ return (Oid) GetNewObjectId8();
+}
+
+/*
+ * GetNewObjectId8 -- allocate a new OID (8 bytes)
+ *
+ * This routine can be called directly if the consumer of the OID allocated
+ * stores the counter in an 8-byte space, where wraparound does not matter.
+ * We still need to care about the wraparound case in the low 32 bits of the
+ * space allocated, GetNewObjectId() expecting OIDs to never be allocated
+ * up to FirstNormalObjectId.
+ */
+Oid8
+GetNewObjectId8(void)
+{
+ Oid8 result;
+ Oid nextoid_lo;
+ uint32 nextoid_hi;
/* safety check, we should never get this far in a HS standby */
if (RecoveryInProgress())
elog(ERROR, "cannot assign OIDs during recovery");
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
+ nextoid_lo = (Oid) TransamVariables->nextOid;
+ nextoid_hi = (uint32) (TransamVariables->nextOid >> 32);
/*
- * Check for wraparound of the OID counter. We *must* not return 0
- * (InvalidOid), and in normal operation we mustn't return anything below
- * FirstNormalObjectId since that range is reserved for initdb (see
- * IsCatalogRelationOid()). Note we are relying on unsigned comparison.
+ * Check for wraparound of the OID counter in its lower 4 bytes. We
+ * *must* not return 0 (InvalidOid), and in normal operation we
+ * mustn't return anything below FirstNormalObjectId since that range
+ * is reserved for initdb (see IsCatalogRelationOid()). Note we are
+ * relying on unsigned comparison.
*
* During initdb, we start the OID generator at FirstGenbkiObjectId, so we
* only wrap if before that point when in bootstrap or standalone mode.
@@ -575,26 +595,32 @@ GetNewObjectId(void)
* available for automatic assignment during initdb, while ensuring they
* will never conflict with user-assigned OIDs.
*/
- if (TransamVariables->nextOid < ((Oid) FirstNormalObjectId))
+ if (nextoid_lo < ((Oid) FirstNormalObjectId))
{
if (IsPostmasterEnvironment)
{
/* wraparound, or first post-initdb assignment, in normal mode */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
else
{
/* we may be bootstrapping, so don't enforce the full range */
- if (TransamVariables->nextOid < ((Oid) FirstGenbkiObjectId))
+ if (nextoid_lo < ((Oid) FirstGenbkiObjectId))
{
/* wraparound in standalone mode (unlikely but possible) */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
}
}
+ /*
+ * Set next OID in its 8-byte space, skipping the first post-init
+ * assignment.
+ */
+ TransamVariables->nextOid = ((Oid8) nextoid_hi) << 32 | nextoid_lo;
+
/* If we run out of logged for use oids then we must log more */
if (TransamVariables->oidCount == 0)
{
@@ -619,7 +645,7 @@ GetNewObjectId(void)
* to the specified value.
*/
static void
-SetNextObjectId(Oid nextOid)
+SetNextObjectId(Oid8 nextOid)
{
/* Safety check, this is only allowable during initdb */
if (IsPostmasterEnvironment)
@@ -629,7 +655,7 @@ SetNextObjectId(Oid nextOid)
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
if (TransamVariables->nextOid > nextOid)
- elog(ERROR, "too late to advance OID counter to %u, it is now %u",
+ elog(ERROR, "too late to advance OID counter to " OID8_FORMAT ", it is now " OID8_FORMAT,
nextOid, TransamVariables->nextOid);
TransamVariables->nextOid = nextOid;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 98c83be84ec9..a970bca6f649 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8564,10 +8564,10 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
* Write a NEXTOID log record
*/
void
-XLogPutNextOid(Oid nextOid)
+XLogPutNextOid(Oid8 nextOid)
{
XLogBeginInsert();
- XLogRegisterData(&nextOid, sizeof(Oid));
+ XLogRegisterData(&nextOid, sizeof(Oid8));
(void) XLogInsert(RM_XLOG_ID, XLOG_NEXTOID);
/*
@@ -8832,7 +8832,7 @@ xlog_redo(XLogReaderState *record)
if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
/*
* We used to try to take the maximum of TransamVariables->nextOid and
@@ -8841,7 +8841,7 @@ xlog_redo(XLogReaderState *record)
* anyway, better to just believe the record exactly. We still take
* OidGenLock while setting the variable, just in case.
*/
- memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid));
+ memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid8));
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
TransamVariables->nextOid = nextOid;
TransamVariables->oidCount = 0;
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 73b78a83fa74..738a1edd2b00 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -829,7 +829,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
LSN_FORMAT_ARGS(checkPoint.redo),
wasShutdown ? "true" : "false"));
ereport(DEBUG1,
- (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: %u",
+ (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: " OID8_FORMAT,
U64FromFullTransactionId(checkPoint.nextXid),
checkPoint.nextOid)));
ereport(DEBUG1,
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index fe5fc5ec133b..a5f53af9367b 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -267,7 +267,7 @@ main(int argc, char *argv[])
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile->checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile->checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 571805cbd54c..cdf9499d6c47 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -82,7 +82,7 @@ static TransactionId oldest_commit_ts_xid_val;
static TransactionId newest_commit_ts_xid_val;
static bool next_oid_given = false;
-static Oid next_oid_val;
+static Oid8 next_oid_val;
static bool mxids_given = false;
static MultiXactId next_mxid_val;
@@ -253,7 +253,7 @@ main(int argc, char *argv[])
case 'o':
errno = 0;
- next_oid_val = strtouint32_strict(optarg, &endptr, 0);
+ next_oid_val = strtou64(optarg, &endptr, 0);
if (endptr == optarg || *endptr != '\0' || errno != 0)
{
pg_log_error("invalid argument for option %s", "-o");
@@ -771,7 +771,7 @@ PrintControlValues(bool guessed)
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile.checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile.checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile.checkPointCopy.nextMulti);
@@ -857,7 +857,7 @@ PrintNewControlValues(void)
if (next_oid_given)
{
- printf(_("NextOID: %u\n"),
+ printf(_("NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
}
@@ -1232,7 +1232,7 @@ usage(void)
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
- printf(_(" -o, --next-oid=OID set next OID\n"));
+ printf(_(" -o, --next-oid=OID8 set next OID (8 bytes)\n"));
printf(_(" -O, --multixact-offset=OFFSET set next multitransaction offset\n"));
printf(_(" -u, --oldest-transaction-id=XID set oldest transaction ID\n"));
printf(_(" -x, --next-transaction-id=XID set next transaction ID\n"));
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index 41f2b1d480c5..83483d883bd1 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -282,11 +282,11 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>-o <replaceable class="parameter">oid</replaceable></option></term>
- <term><option>--next-oid=<replaceable class="parameter">oid</replaceable></option></term>
+ <term><option>-o <replaceable class="parameter">oid8</replaceable></option></term>
+ <term><option>--next-oid=<replaceable class="parameter">oid8</replaceable></option></term>
<listitem>
<para>
- Manually set the next OID.
+ Manually set the next OID (8 bytes).
</para>
<para>
--
2.54.0
v1-0008-Add-relation-option-toast_value_type.patch (text/plain, 6.5K) download
From c7b99583bfc4155ec4c80710deb14f7b463a6537 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:54:58 +0900
Subject: [PATCH v1 08/11] Add relation option toast_value_type
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is created
initially. This parameter has no effect if a TOAST table exists, even
after it is modified later on, even on rewrites.
This can be set only to "oid" currently, and will be expanded with a
second mode later.
Note: perhaps it would make sense to introduce that only when support
for 8-byte OID values are added to TOAST, the split is here to ease
review.
---
src/include/utils/rel.h | 17 +++++++++++++++++
src/backend/access/common/reloptions.c | 21 +++++++++++++++++++++
src/backend/catalog/toasting.c | 6 ++++++
src/bin/psql/tab-complete.in.c | 1 +
doc/src/sgml/ref/create_table.sgml | 18 ++++++++++++++++++
5 files changed, 63 insertions(+)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index cd1e92f23025..f3be0acd4255 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -340,11 +340,20 @@ typedef enum StdRdOptIndexCleanup
STDRD_OPTION_VACUUM_INDEX_CLEANUP_ON,
} StdRdOptIndexCleanup;
+/* StdRdOptions->toast_value_type values */
+typedef enum StdRdOptToastValueType
+{
+ STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+} StdRdOptToastValueType;
+
typedef struct StdRdOptions
{
int32 vl_len_; /* varlena header (do not touch directly!) */
int fillfactor; /* page fill factor in percent (0..100) */
int toast_tuple_target; /* target for tuple toasting */
+ StdRdOptToastValueType toast_value_type; /* type assigned to chunk_id
+ * at toast table creation */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
int parallel_workers; /* max number of parallel workers */
@@ -369,6 +378,14 @@ typedef struct StdRdOptions
((relation)->rd_options ? \
((StdRdOptions *) (relation)->rd_options)->toast_tuple_target : (defaulttarg))
+/*
+ * RelationGetToastValueType
+ * Returns the relation's toast_value_type. Note multiple eval of argument!
+ */
+#define RelationGetToastValueType(relation, defaulttarg) \
+ ((relation)->rd_options ? \
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)
+
/*
* RelationGetFillFactor
* Returns the relation's fillfactor. Note multiple eval of argument!
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 3e832c3797e8..66fd2a05538e 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -548,6 +548,14 @@ static relopt_enum_elt_def viewCheckOptValues[] =
{(const char *) NULL} /* list terminator */
};
+/* values from StdRdOptToastValueType */
+static relopt_enum_elt_def StdRdOptToastValueTypes[] =
+{
+ /* no value for INVALID */
+ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {(const char *) NULL} /* list terminator */
+};
+
static relopt_enum enumRelOpts[] =
{
{
@@ -561,6 +569,17 @@ static relopt_enum enumRelOpts[] =
STDRD_OPTION_VACUUM_INDEX_CLEANUP_AUTO,
gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")
},
+ {
+ {
+ "toast_value_type",
+ "Controls the attribute type of chunk_id at toast table creation",
+ RELOPT_KIND_HEAP,
+ ShareUpdateExclusiveLock
+ },
+ StdRdOptToastValueTypes,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ gettext_noop("Valid values are \"oid\".")
+ },
{
{
"buffering",
@@ -2008,6 +2027,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind)
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_analyze_min_duration)},
{"toast_tuple_target", RELOPT_TYPE_INT,
offsetof(StdRdOptions, toast_tuple_target)},
+ {"toast_value_type", RELOPT_TYPE_ENUM,
+ offsetof(StdRdOptions, toast_value_type)},
{"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL,
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)},
{"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL,
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index bc0a72268335..28ddbd10d357 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -159,9 +159,15 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!IsBinaryUpgrade)
{
+ StdRdOptToastValueType value_type;
+
/* Normal mode, normal check */
if (!needs_toast_table(rel))
return false;
+
+ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
+ if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
+ toast_chunkid_typid = OIDOID;
}
else
{
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index db65d130fcbb..8f6d448e8df8 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -1463,6 +1463,7 @@ static const char *const table_storage_parameters[] = {
"toast.vacuum_max_eager_freeze_failure_rate",
"toast.vacuum_truncate",
"toast_tuple_target",
+ "toast_value_type",
"user_catalog_table",
"vacuum_index_cleanup",
"vacuum_max_eager_freeze_failure_rate",
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index e342585c7f08..3da754045c04 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1653,6 +1653,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry id="reloption-toast-value-type" xreflabel="toast_value_type">
+ <term><literal>toast_value_type</literal> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>toast_value_type</varname> storage parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ The toast_value_type specifies the attribute type of
+ <literal>chunk_id</literal> used when initially creating a toast
+ relation for this table.
+ By default this parameter is <literal>oid</literal>, to assign
+ <type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter cannot be set for TOAST tables.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="reloption-parallel-workers" xreflabel="parallel_workers">
<term><literal>parallel_workers</literal> (<type>integer</type>)
<indexterm>
--
2.54.0
v1-0009-Add-support-for-oid8-TOAST-values.patch (text/plain, 17.1K) download
From 37881f01b5e30d75771e8f0888b668d368f64b20 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 17:06:10 +0900
Subject: [PATCH v1 09/11] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag, with
all the values inserted in the bigint TOAST tables fed from the existing
OID value generator. This will be changed in an upcoming patch that
adds more vartag_external types and its associated structures, with the
code being able to use a different external TOAST pointer depending on
the attribute type of chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 1 +
src/backend/access/common/toast_internals.c | 94 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 19 ++++-
src/backend/catalog/toasting.c | 24 +++++-
doc/src/sgml/ref/create_table.sgml | 2 +
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 22 +++--
9 files changed, 132 insertions(+), 41 deletions(-)
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f3be0acd4255..3c64991533d8 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -345,6 +345,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 66fd2a05538e..fda776215cdd 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..e9ced27baf7e 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,8 +132,10 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -200,24 +203,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or int8-based TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OIDOID)
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ Assert(false);
}
else
{
@@ -263,17 +274,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OIDOID)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
}
}
@@ -303,7 +319,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OIDOID)
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ else if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -384,6 +403,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -395,6 +415,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -405,10 +427,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +485,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +493,24 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..a93e98788da7 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +658,18 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 28ddbd10d357..66d483bb1be3 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -168,6 +169,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
toast_chunkid_typid = OIDOID;
+ else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)
+ toast_chunkid_typid = OID8OID;
}
else
{
@@ -200,7 +203,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -225,6 +229,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -343,7 +360,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OIDOID)
+ opclassIds[0] = OID_BTREE_OPS_OID;
+ else if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3da754045c04..3ed8f3581177 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1666,6 +1666,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
relation for this table.
By default this parameter is <literal>oid</literal>, to assign
<type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter can be set to <type>oid8</type> to use <type>oid8</type>
+ as attribute type for <literal>chunk_id</literal>.
This parameter cannot be set for TOAST tables.
</para>
</listitem>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 6b1e00bfd38e..7afe435747d2 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an 8-byte integer identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 119a54a9404c..ad6c37eefae6 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1878,7 +1878,11 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
Oid8 toast_valueid;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1886,10 +1890,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.54.0
v1-0010-Add-tests-for-TOAST-relations-with-oid8-as-value-.patch (text/plain, 21.5K) download
From d639553a898c548cb745f4f240a74f8edf2ad494 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 May 2026 10:20:36 +0900
Subject: [PATCH v1 10/11] Add tests for TOAST relations with oid8 as value
type
This adds coverage for relations created with default_toast_type =
'oid8', for external TOAST pointers both compressed and uncompressed.
---
src/test/regress/expected/strings.out | 231 ++++++++++++++++++----
src/test/regress/expected/type_sanity.out | 6 +-
src/test/regress/sql/strings.sql | 134 +++++++++----
src/test/regress/sql/type_sanity.sql | 6 +-
4 files changed, 296 insertions(+), 81 deletions(-)
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index a49b75fa1f99..daf8507853d8 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,72 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
substr
--------
567890
@@ -2105,7 +2263,6 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index 1d21d3eb4467..4fabe08f74c0 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -578,15 +578,15 @@ WHERE c1.relnatts != (SELECT count(*) FROM pg_attribute AS a1
(0 rows)
-- Cross-check against pg_type entry
--- NOTE: we allow attstorage to be 'plain' even when typstorage is not;
--- this is mainly for toast tables.
+-- NOTE: we allow attstorage to be 'plain' or 'external' even when typstorage
+-- is not; this is mainly for toast tables.
SELECT a1.attrelid, a1.attname, t1.oid, t1.typname
FROM pg_attribute AS a1, pg_type AS t1
WHERE a1.atttypid = t1.oid AND
(a1.attlen != t1.typlen OR
a1.attalign != t1.typalign OR
a1.attbyval != t1.typbyval OR
- (a1.attstorage != t1.typstorage AND a1.attstorage != 'p'));
+ (a1.attstorage != t1.typstorage AND a1.attstorage NOT IN ('e', 'p')));
attrelid | attname | oid | typname
----------+---------+-----+---------
(0 rows)
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 5ae0e7da31a3..33227d89fac7 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,147 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-- test internally compressing datums
diff --git a/src/test/regress/sql/type_sanity.sql b/src/test/regress/sql/type_sanity.sql
index 95d5b6e09151..5c166f00bb7e 100644
--- a/src/test/regress/sql/type_sanity.sql
+++ b/src/test/regress/sql/type_sanity.sql
@@ -420,8 +420,8 @@ WHERE c1.relnatts != (SELECT count(*) FROM pg_attribute AS a1
WHERE a1.attrelid = c1.oid AND a1.attnum > 0);
-- Cross-check against pg_type entry
--- NOTE: we allow attstorage to be 'plain' even when typstorage is not;
--- this is mainly for toast tables.
+-- NOTE: we allow attstorage to be 'plain' or 'external' even when typstorage
+-- is not; this is mainly for toast tables.
SELECT a1.attrelid, a1.attname, t1.oid, t1.typname
FROM pg_attribute AS a1, pg_type AS t1
@@ -429,7 +429,7 @@ WHERE a1.atttypid = t1.oid AND
(a1.attlen != t1.typlen OR
a1.attalign != t1.typalign OR
a1.attbyval != t1.typbyval OR
- (a1.attstorage != t1.typstorage AND a1.attstorage != 'p'));
+ (a1.attstorage != t1.typstorage AND a1.attstorage NOT IN ('e', 'p')));
-- Look for IsCatalogTextUniqueIndexOid() omissions.
--
2.54.0
v1-0011-Add-support-for-64-bit-TOAST-pointers.patch (text/plain, 42.9K) download
From e69450b3ee2511ea36552b0b065387280f5b52c3 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 May 2026 14:58:10 +0900
Subject: [PATCH v1 11/11] Add support for 64-bit TOAST pointers
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 3 +
src/include/access/heaptoast.h | 16 +-
src/include/varatt.h | 78 ++++-
src/backend/access/common/detoast.c | 171 +++++++---
src/backend/access/common/toast_internals.c | 316 ++++++++++++------
src/backend/access/heap/heaptoast.c | 5 +-
.../replication/logical/reorderbuffer.c | 44 ++-
doc/src/sgml/storage.sgml | 10 +-
contrib/amcheck/verify_heapam.c | 102 ++++--
9 files changed, 537 insertions(+), 208 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..f065ea034e98 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -30,6 +30,9 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..b7ef69a8f56b 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,13 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 33979bbaaad1..6d7ed246fe2f 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -38,6 +38,34 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except that va_valueid is split into two uint32 halves to enforce a
+ * stricter structure size without padding.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+/* Helper macros to get/set the 64-bit value ID from the lo/hi fields */
+#define VARATT_EXTERNAL_OID8_GET_VALUEID(tp) \
+ ((Oid8) (tp).va_valueid_lo | ((Oid8) (tp).va_valueid_hi << 32))
+#define VARATT_EXTERNAL_OID8_SET_VALUEID(tp, id) \
+ do { \
+ (tp).va_valueid_lo = (uint32) (id); \
+ (tp).va_valueid_hi = (uint32) ((id) >> 32); \
+ } while (0)
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -87,6 +115,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -108,6 +137,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -361,7 +392,19 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
+}
+
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
+{
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
}
/* Is varlena datum an indirect pointer? */
@@ -519,16 +562,6 @@ VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
* Testing whether an externally-stored value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -543,4 +576,27 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid8.
+ */
+static inline Size
+VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+}
+
+static inline uint32
+VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+}
+
+static inline bool
+VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)
+{
+ return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <
+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+}
+
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..e1919809751c 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,40 +225,51 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /* Extract extsize and compression info from the appropriate pointer */
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
- * For compressed values, we need to fetch enough slices to decompress
- * at least the requested part (when a prefix is requested).
- * Otherwise, just fetch all slices.
+ * For compressed values, we need to fetch enough slices to
+ * decompress at least the requested part (when a prefix is
+ * requested). Otherwise, just fetch all slices.
+ *
+ * At least for now, if it's LZ4 data, we'll have to fetch the
+ * whole thing, because there doesn't seem to be an API call to
+ * determine how much compressed data we need to be sure of being
+ * able to decompress the required slice.
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
- /*
- * Determine maximum amount of compressed data needed for a prefix
- * of a given length (after decompression).
- *
- * At least for now, if it's LZ4 data, we'll have to fetch the
- * whole thing, because there doesn't seem to be an API call to
- * determine how much compressed data we need to be sure of being
- * able to decompress the required slice.
- */
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,23 +355,47 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- result = (varlena *) palloc(attrsize + VARHDRSZ);
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
+
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -369,10 +404,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +433,43 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +482,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +495,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +504,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +605,20 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +675,20 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index e9ced27baf7e..a4d56ae3a967 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -125,7 +125,6 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -133,9 +132,17 @@ toast_save_datum(Relation rel, Datum value,
int num_indexes;
int validIndex;
Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
Assert(OidIsValid(toast_typid));
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -165,28 +172,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -198,13 +209,14 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or int8-based TOAST relations.
+ * Choose a new value to use as the value ID for this toast value, and
+ * determine the maximum chunk size based on the TOAST table's chunk_id
+ * type.
*
* Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
@@ -213,84 +225,125 @@ toast_save_datum(Relation rel, Datum value,
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * conflict with either new or existing toast value IDs. For Oid8 tables,
+ * value conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OIDOID)
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
+ if (!OidIsValid(rel->rd_toastoid))
+ {
+ /* normal case: just choose a new Oid8 */
+ va_valueid = GetNewObjectId8();
+ }
else
- Assert(false);
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid8;
+ if (oldexternal != NULL)
+ {
+ varatt_external_oid8 old_toast_pointer;
+
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /* This value came from the old toast table; reuse its ID */
+ va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);
+
+ /*
+ * Check if this value already exists in the new toast
+ * table (corner case during table rewrite with multiple
+ * versions of the same row).
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
+ }
+ }
+ if (va_valueid == InvalidOid8)
+ {
+ /*
+ * New value. For Oid8 we just pick a new ID without worrying
+ * about conflicts.
+ */
+ va_valueid = GetNewObjectId8();
+ }
+ }
+
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
+ if (!OidIsValid(rel->rd_toastoid))
{
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ /* normal case: just choose an unused OID */
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
+ else
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid;
+ if (oldexternal != NULL)
{
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
+ varatt_external_oid old_toast_pointer;
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
{
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
+ /* This value came from the old toast table; reuse its OID */
+ va_valueid = old_toast_pointer.va_valueid;
+
+ /*
+ * There is a corner case here: the table rewrite might
+ * have to copy both live and recently-dead versions of a
+ * row, and those versions could easily reference the same
+ * toast value. When we copy the second or later version
+ * of such a row, reusing the OID will mean we select an
+ * OID that's already in the new toast table. Check for
+ * that, and if so, just fall through without writing the
+ * data again.
+ *
+ * While annoying and ugly-looking, this is a good thing
+ * because it ensures that we wind up with only one copy
+ * of the toast value when there is only one copy in the
+ * old toast table. Before we detected this case, we'd
+ * have made multiple copies, wasting space; and what's
+ * worse, the copies belonging to already-deleted heap
+ * tuples would not be reclaimed by VACUUM.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
}
}
- }
- if (toast_pointer.va_valueid == InvalidOid)
- {
- /*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
- */
- if (toast_typid == OIDOID)
+ if (va_valueid == InvalidOid)
{
+ /*
+ * new value; must choose a value that doesn't conflict in
+ * either old or new toast table.
+ */
do
{
- toast_pointer.va_valueid =
+ va_valueid =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ va_valueid));
}
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
}
+
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
/*
@@ -314,15 +367,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
- if (toast_typid == OIDOID)
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
- else if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -376,11 +429,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -395,7 +471,6 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -403,42 +478,75 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toast_typid;
+ Oid toastrelid;
+ vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
/*
- * Open the toast relation and its indexes
+ * Determine the pointer type from the datum's vartag and extract the
+ * toast relation OID and value ID accordingly. The vartag tells us
+ * everything we need — no TOAST table schema lookup required.
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
- toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ tag = VARTAG_EXTERNAL(attr);
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
+ if (tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- if (toast_typid == OIDOID)
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toastrelid = toast_pointer8.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ Assert(tag == VARTAG_ONDISK_OID);
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toastrelid = toast_pointer.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(toast_pointer.va_valueid));
- else
- Assert(false);
+ }
/*
* Find all the chunks. (We don't actually care whether we see them in
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index a93e98788da7..364aec11fc88 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -650,7 +650,10 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ if (toast_typid == OID8OID)
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+ else
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a7b524906995..6c71fcee5816 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5135,12 +5135,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* va_rawsize is the size of the original datum -- including header */
varatt_external_oid toast_pointer;
+ varatt_external_oid8 toast_pointer8;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
+ int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5160,8 +5162,19 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (!VARATT_IS_EXTERNAL(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ /* Branch on vartag to handle both pointer types */
+ if (VARTAG_EXTERNAL(varlena_pointer) == VARTAG_ONDISK_OID8)
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);
+ toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ rawsize = toast_pointer8.va_rawsize;
+ }
+ else
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
+ rawsize = toast_pointer.va_rawsize;
+ }
/*
* Check whether the toast tuple changed, replace if so.
@@ -5179,7 +5192,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(rawsize);
ent->reconstructed = reconstructed;
@@ -5204,13 +5217,28 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ /* Verify size and set compression status based on pointer type */
+ if (VARTAG_EXTERNAL(varlena_pointer) == VARTAG_ONDISK_OID8)
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 7afe435747d2..996c2f26d737 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,9 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an 8-byte integer, regardless of the actual size of the represented
+value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index ad6c37eefae6..0e1c83958af3 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1566,9 +1568,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
Oid8 toast_valueid;
int32 max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
+ toast_valueid = ta->va_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
/* Sanity-check the sequence number. */
@@ -1673,8 +1677,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1733,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1778,26 +1784,46 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
/*
- * Must copy attr into toast_pointer for alignment considerations
+ * Must copy attr into toast_pointer for alignment considerations.
+ * Branch on the tag to determine which pointer type to extract.
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ va_tag_value = VARTAG_EXTERNAL(attr);
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ va_rawsize = toast_pointer8.va_rawsize;
+ va_extinfo = toast_pointer8.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
+ va_rawsize = toast_pointer.va_rawsize;
+ va_extinfo = toast_pointer.va_extinfo;
+ }
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1851,7 +1877,23 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ ta->tag = va_tag_value;
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 tp;
+
+ VARATT_EXTERNAL_GET_POINTER(tp, attr);
+ ta->va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(tp);
+ ta->va_extinfo = tp.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid tp;
+
+ VARATT_EXTERNAL_GET_POINTER(tp, attr);
+ ta->va_valueid = tp.va_valueid;
+ ta->va_extinfo = tp.va_extinfo;
+ }
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1879,29 +1921,33 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
Oid8 toast_valueid;
int32 max_chunk_size;
- Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
+ if (ta->tag == VARTAG_ONDISK_OID8)
+ {
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ ObjectId8GetDatum(toast_valueid));
+ }
else
- Assert(false);
+ {
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) toast_valueid));
+ }
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1921,8 +1967,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
--
2.54.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-30 04:38 Yugo Nagata <nagata@sraoss.co.jp>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Yugo Nagata @ 2026-07-30 04:38 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Fri, 8 May 2026 15:07:13 +0900
Michael Paquier <michael@paquier.xyz> wrote:
> Hi all,
>
> This is a follow-up of the previous thread about $subject, with a
> reworked patch set for discussion in v20, as I care about the subject:
> https://www.postgresql.org/message-id/aFOnKHG7Wn-Srnpv@paquier.xyz
>
> The main feedback of the previous thread is that the previous
> implementation with its callbacks for each vartag was not liked much,
> and their were concerns with pointer redirections and performance.
> This patch set uses what I am calling the "brutal" approach, relying
> on a vartag_external of a varlena or the atttype of the TOAST relation
> to decide which external toast pointer we should use. This uses no
> function pointers, and patches the code to deal with Oid or Oid8 TOAST
> values where it matters. So, this time, performance cannot really be
> an issue.
>
> The patch set is structured so as all the ground work happens first
> (most of it comes from the previous patch set, reorganized a bit), and
> the introduction of the varatt pieces are last, based on the following
> rules:
> - A table can use a 8-byte TOAST value with a new reloption, named
> toast_value_type that can be set to "oid" (default) or "oid8",
> creating a TOAST table with a value of the assigned type. This
> includes support for dumps as well as binary upgrades, so as the
> atttype of the chunk_id of the TOAST table is preserved. A table with
> a TOAST type assigned cannot be changed to a different type through a
> VACUUM FULL or a rewrite, as a matter of implementation simplicity.
> - Renames and cleanup of various areas related to varatt_external,
> renaming things to use OID.
> - The 8-byte TOAST values rely on a Oid8, whose value is retrieved
> from the control file extended by 4 bytes. The code supports
> wraparound of values so as we don't assign anything between 0 and
> FirstNormalObjectId for the lower bytes, same way as before.
>
> The last patch introduces a new vartag_external and the new
> varatt_external_oid8, with an Oid8 as value. Well, not exactly, the
> patch uses two uint32 fields so as the structure is packed without
> padding, as of:
> typedef struct varatt_external_oid8
> {
> int32 va_rawsize;
> uint32 va_extinfo;
> uint32 va_valueid_lo;
> uint32 va_valueid_hi;
> Oid va_toastrelid;
> } varatt_external_oid8;
>
> Note that if applying all the patches except the last one, the code
> would use an varatt_external_oid with an oid8 TOAST table. This
> works, the split is to make reviews easier. An oid8 TOAST table
> always uses a varatt_external_oid8.
>
> I have done a lot of back-and-forth in the patch to try to find a good
> balance between the manipulation of the varlenas in the detoast and
> compression paths, as well as reorderbuffer.c and amcheck. And I have
> finished with the attached, which is kind of nice. The last patch has
> a low footprint:
> 9 files changed, 537 insertions(+), 208 deletions(-)
I applied the patches to the master branch and run a simple test.
After running:
$ pg_resetwal -o 4300000000 -D data
CREATE TABLE tbl_oid8(t text) with (toast_value_type = 'oid8');
ALTER TABLE tbl_oid8 ALTER t SET STORAGE EXTERNAL;
CREATE TABLE tbl_oid(t text) with (toast_value_type = 'oid');
ALTER TABLE tbl_oid ALTER t SET STORAGE EXTERNAL;
INSERT INTO tbl_oid8 VALUES (repeat(md5('x'),100));
INSERT INTO tbl_oid VALUES (repeat(md5('x'),100));
SELECT tableoid, t = repeat(md5('x'),100) FROM tbl_oid8;
tableoid | ?column?
----------+----------
5032704 | t
(1 row)
SELECT chunk_id, chunk_seq FROM pg_toast.pg_toast_5032704;
chunk_id | chunk_seq
------------+-----------
4300000010 | 0
4300000010 | 1
(2 rows)
SELECT tableoid, t = repeat(md5('x'),100) FROM tbl_oid;
tableoid | ?column?
----------+----------
5032709 | t
(1 row)
SELECT chunk_id, chunk_seq FROM pg_toast.pg_toast_5032709;
chunk_id | chunk_seq
----------+-----------
5032715 | 0
5032715 | 1
(2 rows)
This appears to work as expected.
I noticed one issue, though.
When toast_value_type is specified as something other than oid or oid8, the error
message is:
postgres=# CREATE TABLE tbl(t text) with (toast_value_type = 'x');
ERROR: invalid value for enum option "toast_value_type": x
DETAIL: Valid values are "oid".
The detail message should be updated to include oid8.
I also have a few small comments on the patches.
- 0009
(1)
The commit message says:
TOAST pointers still rely on varatt_external and a single vartag, with
all the values inserted in the bigint TOAST tables fed from the existing
OID value generator.
This made me a bit confused because chunk_ids insereted into TOAST tables using oid8
are generated by GetNewObjectId8().
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
This value is later cast to Oid here, so what is actually stored is a 4-byte chunk_id,
but saying that the values are "fed from the existing OID value generator" seems
a bit inaccurate to me.
(2)
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or int8-based TOAST relations.
This patch relies on oid8, introduced by b139bd3b6ef, rather than int8, so, I
wonder if this comment sould also be updated.
(3)
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an 8-byte integer identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
Similarly, I think the documentation should mention oid8 rather than 8-byte
integer. That would also be consistent with the CREATE TABLE documentation for
the toast_value_type parameter.
- 0011
(4)
+ /*
+ * Check if this value already exists in the new toast
+ * table (corner case during table rewrite with multiple
+ * versions of the same row).
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
The same code appears later, together with a more detailed comment. How about
moving the detailed comment here instead, or simply referring to the later comment?
(5)
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
+{
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
}
This macro is defined but not used. Is it intended to be used in places that
check VARTAG_ONDISK_OID8 directly, for example:
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
?
(6)
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an 8-byte integer, regardless of the actual size of the represented
+value.
</para>
Should this also refer to Oid8 rather than 8-byte integner, for consistency?
Regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-30 08:01 Yugo Nagata <nagata@sraoss.co.jp>
parent: Yugo Nagata <nagata@sraoss.co.jp>
0 siblings, 2 replies; 72+ messages in thread
From: Yugo Nagata @ 2026-07-30 08:01 UTC (permalink / raw)
To: Yugo Nagata <nagata@sraoss.co.jp>; +Cc: Michael Paquier <michael@paquier.xyz>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Thu, 30 Jul 2026 13:38:18 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
> On Fri, 8 May 2026 15:07:13 +0900
> Michael Paquier <michael@paquier.xyz> wrote:
>
> > Hi all,
> >
> > This is a follow-up of the previous thread about $subject, with a
> > reworked patch set for discussion in v20, as I care about the subject:
> > https://www.postgresql.org/message-id/aFOnKHG7Wn-Srnpv@paquier.xyz
> >
> > The main feedback of the previous thread is that the previous
> > implementation with its callbacks for each vartag was not liked much,
> > and their were concerns with pointer redirections and performance.
> > This patch set uses what I am calling the "brutal" approach, relying
> > on a vartag_external of a varlena or the atttype of the TOAST relation
> > to decide which external toast pointer we should use. This uses no
> > function pointers, and patches the code to deal with Oid or Oid8 TOAST
> > values where it matters. So, this time, performance cannot really be
> > an issue.
> >
> > The patch set is structured so as all the ground work happens first
> > (most of it comes from the previous patch set, reorganized a bit), and
> > the introduction of the varatt pieces are last, based on the following
> > rules:
> > - A table can use a 8-byte TOAST value with a new reloption, named
> > toast_value_type that can be set to "oid" (default) or "oid8",
> > creating a TOAST table with a value of the assigned type. This
> > includes support for dumps as well as binary upgrades, so as the
> > atttype of the chunk_id of the TOAST table is preserved. A table with
> > a TOAST type assigned cannot be changed to a different type through a
> > VACUUM FULL or a rewrite, as a matter of implementation simplicity.
> > - Renames and cleanup of various areas related to varatt_external,
> > renaming things to use OID.
> > - The 8-byte TOAST values rely on a Oid8, whose value is retrieved
> > from the control file extended by 4 bytes. The code supports
> > wraparound of values so as we don't assign anything between 0 and
> > FirstNormalObjectId for the lower bytes, same way as before.
> >
> > The last patch introduces a new vartag_external and the new
> > varatt_external_oid8, with an Oid8 as value. Well, not exactly, the
> > patch uses two uint32 fields so as the structure is packed without
> > padding, as of:
> > typedef struct varatt_external_oid8
> > {
> > int32 va_rawsize;
> > uint32 va_extinfo;
> > uint32 va_valueid_lo;
> > uint32 va_valueid_hi;
> > Oid va_toastrelid;
> > } varatt_external_oid8;
> >
> > Note that if applying all the patches except the last one, the code
> > would use an varatt_external_oid with an oid8 TOAST table. This
> > works, the split is to make reviews easier. An oid8 TOAST table
> > always uses a varatt_external_oid8.
> >
> > I have done a lot of back-and-forth in the patch to try to find a good
> > balance between the manipulation of the varlenas in the detoast and
> > compression paths, as well as reorderbuffer.c and amcheck. And I have
> > finished with the attached, which is kind of nice. The last patch has
> > a low footprint:
> > 9 files changed, 537 insertions(+), 208 deletions(-)
>
> I applied the patches to the master branch and run a simple test.
>
> After running:
>
> $ pg_resetwal -o 4300000000 -D data
>
> CREATE TABLE tbl_oid8(t text) with (toast_value_type = 'oid8');
> ALTER TABLE tbl_oid8 ALTER t SET STORAGE EXTERNAL;
> CREATE TABLE tbl_oid(t text) with (toast_value_type = 'oid');
> ALTER TABLE tbl_oid ALTER t SET STORAGE EXTERNAL;
>
> INSERT INTO tbl_oid8 VALUES (repeat(md5('x'),100));
> INSERT INTO tbl_oid VALUES (repeat(md5('x'),100));
>
> SELECT tableoid, t = repeat(md5('x'),100) FROM tbl_oid8;
> tableoid | ?column?
> ----------+----------
> 5032704 | t
> (1 row)
>
> SELECT chunk_id, chunk_seq FROM pg_toast.pg_toast_5032704;
> chunk_id | chunk_seq
> ------------+-----------
> 4300000010 | 0
> 4300000010 | 1
> (2 rows)
>
> SELECT tableoid, t = repeat(md5('x'),100) FROM tbl_oid;
> tableoid | ?column?
> ----------+----------
> 5032709 | t
> (1 row)
>
> SELECT chunk_id, chunk_seq FROM pg_toast.pg_toast_5032709;
> chunk_id | chunk_seq
> ----------+-----------
> 5032715 | 0
> 5032715 | 1
> (2 rows)
>
> This appears to work as expected.
I also run a simple performance test. I measured the TPS on
the master branch and with the current patches applied under the
following conditions:
Prepare:
drop table if exists tbl;
create table tbl (id int primary key, j int, t text);
alter table tbl alter column t set storage external;
insert into tbl select i, i, repeat(md5(i::text),100)
from generate_series(1,10000) i;
query.sql:
\set id random(1, 10000)
update tbl set (j,t) = (j + 1, repeat(md5((j+1)::text),100))
where id = :id;
Running pbench three times:
$ pgbench test -f query.sql -c 8 -j 4 -T 100
Results:
master: 2686.229104, 2697.655096, 2687.894343
patched: 2685.292256, 2692.080214, 2690.005753
I could not observe any performance degradation with the curernt patches,
at least in my environment.
I haven't tested whether the previous version of the patches showed the
performance degradation that has been a concern, though.
>
> I noticed one issue, though.
> When toast_value_type is specified as something other than oid or oid8, the error
> message is:
>
> postgres=# CREATE TABLE tbl(t text) with (toast_value_type = 'x');
> ERROR: invalid value for enum option "toast_value_type": x
> DETAIL: Valid values are "oid".
>
> The detail message should be updated to include oid8.
>
>
> I also have a few small comments on the patches.
>
> - 0009
>
> (1)
> The commit message says:
>
> TOAST pointers still rely on varatt_external and a single vartag, with
> all the values inserted in the bigint TOAST tables fed from the existing
> OID value generator.
>
> This made me a bit confused because chunk_ids insereted into TOAST tables using oid8
> are generated by GetNewObjectId8().
>
> + else if (toast_typid == OID8OID)
> + toast_pointer.va_valueid = GetNewObjectId8();
>
> This value is later cast to Oid here, so what is actually stored is a 4-byte chunk_id,
> but saying that the values are "fed from the existing OID value generator" seems
> a bit inaccurate to me.
>
> (2)
> /*
> - * Choose an OID to use as the value ID for this toast value.
> + * Choose a new value to use as the value ID for this toast value, be it
> + * for OID or int8-based TOAST relations.
>
> This patch relies on oid8, introduced by b139bd3b6ef, rather than int8, so, I
> wonder if this comment sould also be updated.
>
> (3)
> belonging to the owning table. Every
> -<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
> -identifying the particular <acronym>TOAST</acronym>ed value),
> +<acronym>TOAST</acronym> table has the columns
> +<structfield>chunk_id</structfield> (an OID or an 8-byte integer identifying
> +the particular <acronym>TOAST</acronym>ed value),
> <structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
>
> Similarly, I think the documentation should mention oid8 rather than 8-byte
> integer. That would also be consistent with the CREATE TABLE documentation for
> the toast_value_type parameter.
>
> - 0011
>
> (4)
> + /*
> + * Check if this value already exists in the new toast
> + * table (corner case during table rewrite with multiple
> + * versions of the same row).
> + */
> + if (toastrel_valueid_exists(toastrel, va_valueid))
> + {
> + /* Match, so short-circuit the data storage loop below */
> + data_todo = 0;
> + }
>
> The same code appears later, together with a more detailed comment. How about
> moving the detailed comment here instead, or simply referring to the later comment?
>
> (5)
> +/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
> +static inline bool
> +VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
> +{
> + return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
> }
>
> This macro is defined but not used. Is it intended to be used in places that
> check VARTAG_ONDISK_OID8 directly, for example:
>
> + if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
>
> ?
>
> (6)
>
> any. Allowing for the varlena header bytes,
> -the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
> -bytes regardless of the actual size of the represented value.
> +the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
> +bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
> +when using an 8-byte integer, regardless of the actual size of the represented
> +value.
> </para>
>
> Should this also refer to Oid8 rather than 8-byte integner, for consistency?
I have one additional, very trivial comment:
(7)
/*
- * Open the toast relation and its indexes
+ * Determine the pointer type from the datum's vartag and extract the
+ * toast relation OID and value ID accordingly. The vartag tells us
+ * everything we need $B!=(B no TOAST table schema lookup required.
Non-ASCII hyphen is used in this comment. This may not be prohibitted,
but I don't think it's common in the PostgreSQL code base.
Regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-30 11:16 Hannu Krosing <hannuk@google.com>
parent: Yugo Nagata <nagata@sraoss.co.jp>
1 sibling, 1 reply; 72+ messages in thread
From: Hannu Krosing @ 2026-07-30 11:16 UTC (permalink / raw)
To: Yugo Nagata <nagata@sraoss.co.jp>; +Cc: Michael Paquier <michael@paquier.xyz>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
I am mostly worried about performance degradation caused by the toast index
not fitting in memory.
This is more likely to happen for huge tables, the case for which the
8-byte toast pointer is proposed.
Have you compared the index sizes of OID and 8-byte ID ?
It is possible, that they are of the same size because of alignment, so
there would not be any difference in size.
On Thu, Jul 30, 2026 at 10:01 AM Yugo Nagata <nagata@sraoss.co.jp> wrote:
> On Thu, 30 Jul 2026 13:38:18 +0900
> Yugo Nagata <nagata@sraoss.co.jp> wrote:
>
> > On Fri, 8 May 2026 15:07:13 +0900
> > Michael Paquier <michael@paquier.xyz> wrote:
> >
> > > Hi all,
> > >
> > > This is a follow-up of the previous thread about $subject, with a
> > > reworked patch set for discussion in v20, as I care about the subject:
> > > https://www.postgresql.org/message-id/aFOnKHG7Wn-Srnpv@paquier.xyz
> > >
> > > The main feedback of the previous thread is that the previous
> > > implementation with its callbacks for each vartag was not liked much,
> > > and their were concerns with pointer redirections and performance.
> > > This patch set uses what I am calling the "brutal" approach, relying
> > > on a vartag_external of a varlena or the atttype of the TOAST relation
> > > to decide which external toast pointer we should use. This uses no
> > > function pointers, and patches the code to deal with Oid or Oid8 TOAST
> > > values where it matters. So, this time, performance cannot really be
> > > an issue.
> > >
> > > The patch set is structured so as all the ground work happens first
> > > (most of it comes from the previous patch set, reorganized a bit), and
> > > the introduction of the varatt pieces are last, based on the following
> > > rules:
> > > - A table can use a 8-byte TOAST value with a new reloption, named
> > > toast_value_type that can be set to "oid" (default) or "oid8",
> > > creating a TOAST table with a value of the assigned type. This
> > > includes support for dumps as well as binary upgrades, so as the
> > > atttype of the chunk_id of the TOAST table is preserved. A table with
> > > a TOAST type assigned cannot be changed to a different type through a
> > > VACUUM FULL or a rewrite, as a matter of implementation simplicity.
> > > - Renames and cleanup of various areas related to varatt_external,
> > > renaming things to use OID.
> > > - The 8-byte TOAST values rely on a Oid8, whose value is retrieved
> > > from the control file extended by 4 bytes. The code supports
> > > wraparound of values so as we don't assign anything between 0 and
> > > FirstNormalObjectId for the lower bytes, same way as before.
> > >
> > > The last patch introduces a new vartag_external and the new
> > > varatt_external_oid8, with an Oid8 as value. Well, not exactly, the
> > > patch uses two uint32 fields so as the structure is packed without
> > > padding, as of:
> > > typedef struct varatt_external_oid8
> > > {
> > > int32 va_rawsize;
> > > uint32 va_extinfo;
> > > uint32 va_valueid_lo;
> > > uint32 va_valueid_hi;
> > > Oid va_toastrelid;
> > > } varatt_external_oid8;
> > >
> > > Note that if applying all the patches except the last one, the code
> > > would use an varatt_external_oid with an oid8 TOAST table. This
> > > works, the split is to make reviews easier. An oid8 TOAST table
> > > always uses a varatt_external_oid8.
> > >
> > > I have done a lot of back-and-forth in the patch to try to find a good
> > > balance between the manipulation of the varlenas in the detoast and
> > > compression paths, as well as reorderbuffer.c and amcheck. And I have
> > > finished with the attached, which is kind of nice. The last patch has
> > > a low footprint:
> > > 9 files changed, 537 insertions(+), 208 deletions(-)
> >
> > I applied the patches to the master branch and run a simple test.
> >
> > After running:
> >
> > $ pg_resetwal -o 4300000000 -D data
> >
> > CREATE TABLE tbl_oid8(t text) with (toast_value_type = 'oid8');
> > ALTER TABLE tbl_oid8 ALTER t SET STORAGE EXTERNAL;
> > CREATE TABLE tbl_oid(t text) with (toast_value_type = 'oid');
> > ALTER TABLE tbl_oid ALTER t SET STORAGE EXTERNAL;
> >
> > INSERT INTO tbl_oid8 VALUES (repeat(md5('x'),100));
> > INSERT INTO tbl_oid VALUES (repeat(md5('x'),100));
> >
> > SELECT tableoid, t = repeat(md5('x'),100) FROM tbl_oid8;
> > tableoid | ?column?
> > ----------+----------
> > 5032704 | t
> > (1 row)
> >
> > SELECT chunk_id, chunk_seq FROM pg_toast.pg_toast_5032704;
> > chunk_id | chunk_seq
> > ------------+-----------
> > 4300000010 | 0
> > 4300000010 | 1
> > (2 rows)
> >
> > SELECT tableoid, t = repeat(md5('x'),100) FROM tbl_oid;
> > tableoid | ?column?
> > ----------+----------
> > 5032709 | t
> > (1 row)
> >
> > SELECT chunk_id, chunk_seq FROM pg_toast.pg_toast_5032709;
> > chunk_id | chunk_seq
> > ----------+-----------
> > 5032715 | 0
> > 5032715 | 1
> > (2 rows)
> >
> > This appears to work as expected.
>
> I also run a simple performance test. I measured the TPS on
> the master branch and with the current patches applied under the
> following conditions:
>
> Prepare:
> drop table if exists tbl;
> create table tbl (id int primary key, j int, t text);
> alter table tbl alter column t set storage external;
> insert into tbl select i, i, repeat(md5(i::text),100)
> from generate_series(1,10000) i;
>
> query.sql:
> \set id random(1, 10000)
> update tbl set (j,t) = (j + 1, repeat(md5((j+1)::text),100))
> where id = :id;
>
> Running pbench three times:
>
> $ pgbench test -f query.sql -c 8 -j 4 -T 100
>
> Results:
>
> master: 2686.229104, 2697.655096, 2687.894343
> patched: 2685.292256, 2692.080214, 2690.005753
>
> I could not observe any performance degradation with the curernt patches,
> at least in my environment.
>
> I haven't tested whether the previous version of the patches showed the
> performance degradation that has been a concern, though.
>
> >
> > I noticed one issue, though.
> > When toast_value_type is specified as something other than oid or oid8,
> the error
> > message is:
> >
> > postgres=# CREATE TABLE tbl(t text) with (toast_value_type = 'x');
> > ERROR: invalid value for enum option "toast_value_type": x
> > DETAIL: Valid values are "oid".
> >
> > The detail message should be updated to include oid8.
> >
> >
> > I also have a few small comments on the patches.
> >
> > - 0009
> >
> > (1)
> > The commit message says:
> >
> > TOAST pointers still rely on varatt_external and a single vartag, with
> > all the values inserted in the bigint TOAST tables fed from the existing
> > OID value generator.
> >
> > This made me a bit confused because chunk_ids insereted into TOAST
> tables using oid8
> > are generated by GetNewObjectId8().
> >
> > + else if (toast_typid == OID8OID)
> > + toast_pointer.va_valueid = GetNewObjectId8();
> >
> > This value is later cast to Oid here, so what is actually stored is a
> 4-byte chunk_id,
> > but saying that the values are "fed from the existing OID value
> generator" seems
> > a bit inaccurate to me.
> >
> > (2)
> > /*
> > - * Choose an OID to use as the value ID for this toast value.
> > + * Choose a new value to use as the value ID for this toast value,
> be it
> > + * for OID or int8-based TOAST relations.
> >
> > This patch relies on oid8, introduced by b139bd3b6ef, rather than int8,
> so, I
> > wonder if this comment sould also be updated.
> >
> > (3)
> > belonging to the owning table. Every
> > -<acronym>TOAST</acronym> table has the columns
> <structfield>chunk_id</structfield> (an OID
> > -identifying the particular <acronym>TOAST</acronym>ed value),
> > +<acronym>TOAST</acronym> table has the columns
> > +<structfield>chunk_id</structfield> (an OID or an 8-byte integer
> identifying
> > +the particular <acronym>TOAST</acronym>ed value),
> > <structfield>chunk_seq</structfield> (a sequence number for the chunk
> within its value),
> >
> > Similarly, I think the documentation should mention oid8 rather than
> 8-byte
> > integer. That would also be consistent with the CREATE TABLE
> documentation for
> > the toast_value_type parameter.
> >
> > - 0011
> >
> > (4)
> > + /*
> > + * Check if this value already
> exists in the new toast
> > + * table (corner case during table
> rewrite with multiple
> > + * versions of the same row).
> > + */
> > + if
> (toastrel_valueid_exists(toastrel, va_valueid))
> > + {
> > + /* Match, so short-circuit
> the data storage loop below */
> > + data_todo = 0;
> > + }
> >
> > The same code appears later, together with a more detailed comment. How
> about
> > moving the detailed comment here instead, or simply referring to the
> later comment?
> >
> > (5)
> > +/* Is varlena datum a pointer to on-disk toasted data with 8-byte value
> ID? */
> > +static inline bool
> > +VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
> > +{
> > + return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) ==
> VARTAG_ONDISK_OID8;
> > }
> >
> > This macro is defined but not used. Is it intended to be used in places
> that
> > check VARTAG_ONDISK_OID8 directly, for example:
> >
> > + if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
> >
> > ?
> >
> > (6)
> >
> > any. Allowing for the varlena header bytes,
> > -the total size of an on-disk <acronym>TOAST</acronym> pointer datum is
> therefore 18
> > -bytes regardless of the actual size of the represented value.
> > +the total size of an on-disk <acronym>TOAST</acronym> pointer datum is
> 18
> > +bytes when using an OID as <structfield>chunk_id</structfield>, or 22
> bytes
> > +when using an 8-byte integer, regardless of the actual size of the
> represented
> > +value.
> > </para>
> >
> > Should this also refer to Oid8 rather than 8-byte integner, for
> consistency?
>
> I have one additional, very trivial comment:
>
> (7)
>
> /*
> - * Open the toast relation and its indexes
> + * Determine the pointer type from the datum's vartag and extract
> the
> + * toast relation OID and value ID accordingly. The vartag tells
> us
> + * everything we need ― no TOAST table schema lookup required.
>
> Non-ASCII hyphen is used in this comment. This may not be prohibitted,
> but I don't think it's common in the PostgreSQL code base.
>
> Regards,
> Yugo Nagata
>
> --
> Yugo Nagata <nagata@sraoss.co.jp>
>
>
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-31 02:36 Michael Paquier <michael@paquier.xyz>
parent: Hannu Krosing <hannuk@google.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-07-31 02:36 UTC (permalink / raw)
To: Hannu Krosing <hannuk@google.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Jul 30, 2026 at 01:16:36PM +0200, Hannu Krosing wrote:
> Have you compared the index sizes of OID and 8-byte ID ?
Please do not top-post. This breaks the logical flow of the community
thread. Please see the following about bottom-posting:
https://en.wikipedia.org/wiki/Posting_style#Bottom-posting
> It is possible, that they are of the same size because of alignment, so
> there would not be any difference in size.
I suspect that the size is the same due to 8-byte alignment (not done
a measurement on that with a fixed loading pattern). Just please note
that the OID8 behavior is available on an opt-in as a reloption, with
OID still being the default. So there is no impact by default,
neither is there an impact across upgrades. A relation set with an
OID8 is preserved across pg_upgrade moving forward to newer versions,
of course.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-31 10:58 Hannu Krosing <hannuk@google.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Hannu Krosing @ 2026-07-31 10:58 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Jul 31, 2026 at 4:37 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Jul 30, 2026 at 01:16:36PM +0200, Hannu Krosing wrote:
> > Have you compared the index sizes of OID and 8-byte ID ?
>
> Please do not top-post. This breaks the logical flow of the community
> thread. Please see the following about bottom-posting:
> https://en.wikipedia.org/wiki/Posting_style#Bottom-posting
Sorry about that, I need to get an email client that makes it easier
to see when I'm about to do that :p
> > It is possible, that they are of the same size because of alignment, so
> > there would not be any difference in size.
>
> I suspect that the size is the same due to 8-byte alignment (not done
> a measurement on that with a fixed loading pattern).
I ran a quick test and at least on x64 this is the case indeed:
hannuk=# create table oid_vs_bigint(id oid primary key, int8 bigint unique);
CREATE TABLE
hannuk=# insert into oid_vs_bigint select i, i from
generate_series(1,100000) g(i);
INSERT 0 100000
hannuk=# select indexrelid::regclass, pg_relation_size(indexrelid)
from pg_index where indrelid = 'oid_vs_bigint'::regclass;
indexrelid │ pg_relation_size
────────────────────────┼──────────────────
oid_vs_bigint_pkey │ 2260992
oid_vs_bigint_int8_key │ 2260992
(2 rows)
So, it's not slower because of size, but it's not faster either.
> Just please note
> that the OID8 behavior is available on an opt-in as a reloption, with
> OID still being the default. So there is no impact by default,
> neither is there an impact across upgrades. A relation set with an
> OID8 is preserved across pg_upgrade moving forward to newer versions,
> of course.
Yes, any new TOAST types should live alongside the existing one and
not try to replace them in one go.
My Direct Toast design does the same: either a reloption or a global
GUC for selecting on write and automatic selection by the VARATT type
on reads.
In the case of Direct Toast this also means that both TOAST types can
coexist in the same table for both the main heap and TOAST storage,
and you can switch back and forth at any time, for example, switching
to direct only when you run into the 4B OID limit.
For OID8 I guess you can still have it, but you will need to create a
second oid8 toast table because the toast table format changed in
incompatible ways.
----
Hannu
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-31 11:10 Matthias van de Meent <boekewurm+postgres@gmail.com>
parent: Hannu Krosing <hannuk@google.com>
0 siblings, 1 reply; 72+ messages in thread
From: Matthias van de Meent @ 2026-07-31 11:10 UTC (permalink / raw)
To: Hannu Krosing <hannuk@google.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Fri, 31 Jul 2026 at 12:58, Hannu Krosing <hannuk@google.com> wrote:
>
> I ran a quick test and at least on x64 this is the case indeed:
>
> hannuk=# create table oid_vs_bigint(id oid primary key, int8 bigint unique);
> CREATE TABLE
> hannuk=# insert into oid_vs_bigint select i, i from
> generate_series(1,100000) g(i);
> INSERT 0 100000
> hannuk=# select indexrelid::regclass, pg_relation_size(indexrelid)
> from pg_index where indrelid = 'oid_vs_bigint'::regclass;
> indexrelid │ pg_relation_size
> ────────────────────────┼──────────────────
> oid_vs_bigint_pkey │ 2260992
> oid_vs_bigint_int8_key │ 2260992
> (2 rows)
>
> So, it's not slower because of size, but it's not faster either.
This test is not representative of real toast indexes, because it
skips the chunk number column. The addition of the chunk number column
will impact the size of the index, because now the index items are 16B
for OID vs 24B for OID8, for 20B and 28B page data usage respectively
after taking the 4-byte line pointer into account. Back of the
envelope calculations indicate this means the OID8 index will be about
40% larger.
Kind regards,
Matthias van de Meent.
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-07-31 14:16 Hannu Krosing <hannuk@google.com>
parent: Matthias van de Meent <boekewurm+postgres@gmail.com>
0 siblings, 0 replies; 72+ messages in thread
From: Hannu Krosing @ 2026-07-31 14:16 UTC (permalink / raw)
To: Matthias van de Meent <boekewurm+postgres@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Jul 31, 2026 at 1:10 PM Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
>
...
> This test is not representative of real toast indexes, because it
> skips the chunk number column. The addition of the chunk number column
> will impact the size of the index, because now the index items are 16B
> for OID vs 24B for OID8, for 20B and 28B page data usage respectively
> after taking the 4-byte line pointer into account. Back of the
> envelope calculations indicate this means the OID8 index will be about
> 40% larger.
Yes, you are right:
hannuk=# create table oid_vs_bigint(id oid, int8 bigint, chunk_id int,
unique(id, chunk_id), unique(int8, chunk_id));
CREATE TABLE
hannuk=# insert into oid_vs_bigint select i, i, 0 from
generate_series(1,100000) g(i);
INSERT 0 100000
hannuk=# select indexrelid::regclass, pg_relation_size(indexrelid)
from pg_index where indrelid = 'oid_vs_bigint'::regclass;
indexrelid │ pg_relation_size
─────────────────────────────────┼──────────────────
oid_vs_bigint_id_chunk_id_key │ 2260992
oid_vs_bigint_int8_chunk_id_key │ 3162112
(2 rows)
So indeed 40% more, and you get the *same* number even if you used
OID6 (i.e 6-byte tid)
hannuk=# create table oid_vs_bigint(id oid, int8 tid, chunk_id int,
unique(id, chunk_id), unique(int8, chunk_id));
CREATE TABLE
hannuk=# insert into oid_vs_bigint select i, format('(%s,1)',i)::tid,
0 from generate_series(1,100000) g(i);
INSERT 0 100000
hannuk=# select * from oid_vs_bigint limit 2;
id │ int8 │ chunk_id
────┼───────┼──────────
1 │ (1,1) │ 0
2 │ (2,1) │ 0
(2 rows)
hannuk=# select indexrelid::regclass, pg_relation_size(indexrelid)
from pg_index where indrelid = 'oid_vs_bigint'::regclass;
indexrelid │ pg_relation_size
─────────────────────────────────┼──────────────────
oid_vs_bigint_id_chunk_id_key │ 2260992
oid_vs_bigint_int8_chunk_id_key │ 3162112
(2 rows)
----
Hannu
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-06 04:38 Michael Paquier <michael@paquier.xyz>
parent: Yugo Nagata <nagata@sraoss.co.jp>
1 sibling, 2 replies; 72+ messages in thread
From: Michael Paquier @ 2026-08-06 04:38 UTC (permalink / raw)
To: Yugo Nagata <nagata@sraoss.co.jp>; +Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Jul 30, 2026 at 05:01:02PM +0900, Yugo Nagata wrote:
> On Thu, 30 Jul 2026 13:38:18 +0900 Yugo Nagata <nagata@sraoss.co.jp> wrote:
> I also run a simple performance test. I measured the TPS on
> the master branch and with the current patches applied under the
> following conditions:
>
> Running pbench three times:
>
> $ pgbench test -f query.sql -c 8 -j 4 -T 100
>
> Results:
>
> master: 2686.229104, 2697.655096, 2687.894343
> patched: 2685.292256, 2692.080214, 2690.005753
Thanks for the test.
> I haven't tested whether the previous version of the patches showed the
> performance degradation that has been a concern, though.
I am not sure that it is worth doing; I am not planning to get back to
these versions, but it's true that it could be useful to see if your
environment shows an impact of some impact. That's low priority, IMO.
>> postgres=# CREATE TABLE tbl(t text) with (toast_value_type = 'x');
>> ERROR: invalid value for enum option "toast_value_type": x
>> DETAIL: Valid values are "oid".
>>
>> The detail message should be updated to include oid8.
I had this one already fixed on my local branch.
>> - 0009
>>
>> (1)
>> The commit message says:
>>
>> TOAST pointers still rely on varatt_external and a single vartag, with
>> all the values inserted in the bigint TOAST tables fed from the existing
>> OID value generator.
>>
>> This made me a bit confused because chunk_ids insereted into TOAST tables using oid8
>> are generated by GetNewObjectId8().
>>
>> + else if (toast_typid == OID8OID)
>> + toast_pointer.va_valueid = GetNewObjectId8();
>>
>> This value is later cast to Oid here, so what is actually stored is a 4-byte chunk_id,
>> but saying that the values are "fed from the existing OID value generator" seems
>> a bit inaccurate to me.
Okay, edited that a bit. I tend to rework all my commit messages
before pushing, still you are right that this sounded a bit confusing.
>> (2)
>> /*
>> - * Choose an OID to use as the value ID for this toast value.
>> + * Choose a new value to use as the value ID for this toast value, be it
>> + * for OID or int8-based TOAST relations.
>>
>> This patch relies on oid8, introduced by b139bd3b6ef, rather than int8, so, I
>> wonder if this comment sould also be updated.
Fixed.
>> (3)
>> belonging to the owning table. Every
>> -<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
>> -identifying the particular <acronym>TOAST</acronym>ed value),
>> +<acronym>TOAST</acronym> table has the columns
>> +<structfield>chunk_id</structfield> (an OID or an 8-byte integer identifying
>> +the particular <acronym>TOAST</acronym>ed value),
>> <structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
>>
>> Similarly, I think the documentation should mention oid8 rather than 8-byte
>> integer. That would also be consistent with the CREATE TABLE documentation for
>> the toast_value_type parameter.
Hole in the docs, indeed.
>> - 0011
>>
>> (4)
>> + /*
>> + * Check if this value already exists in the new toast
>> + * table (corner case during table rewrite with multiple
>> + * versions of the same row).
>> + */
>> + if (toastrel_valueid_exists(toastrel, va_valueid))
>> + {
>> + /* Match, so short-circuit the data storage loop below */
>> + data_todo = 0;
>> + }
>>
>> The same code appears later, together with a more detailed comment. How about
>> moving the detailed comment here instead, or simply referring to
>> the later comment?
FWIW, I was looking again at this block of code, and reminded myself
why this piece is this way: I want GetNewObjectId8(), InvalidOid8 and
varatt_external_oid8 to have their own references in each block
depending on the type of chunk_id. Perhaps a bit repetitive,
efficient at least.
>> (5)
>> +/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
>> +static inline bool
>> +VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
>> +{
>> + return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
>> }
>>
>> This macro is defined but not used. Is it intended to be used in places that
>> check VARTAG_ONDISK_OID8 directly, for example:
>>
>> + if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
>>
>> ?
Consistency with the other one, for extensions, as all the code paths
checking for VARTAG_ONDISK_OID8 already make sure that we are dealing
with an external pointer, so I did not see a point in checking
VARATT_IS_EXTERNAL_ONDISK_OID8, that would make the extra
VARATT_IS_EXTERNAL() check pointless.
>> (6)
>>
>> any. Allowing for the varlena header bytes,
>> -the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
>> -bytes regardless of the actual size of the represented value.
>> +the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
>> +bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
>> +when using an 8-byte integer, regardless of the actual size of the represented
>> +value.
>> </para>
>>
>> Should this also refer to Oid8 rather than 8-byte integner, for consistency?
Fixed.
> (7)
>
> Non-ASCII hyphen is used in this comment. This may not be prohibitted,
> but I don't think it's common in the PostgreSQL code base.
Seems like my keyboard has slipped a bit here. Will fix after
replying to your other comments.
--
Michael
Attachments: download all (12 files, .tar.gz)
v2-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-ins.patch (text/plain, 15.3K) download
From 3ef9f1da1f7a9b8849665ba36c1f4356336e53cb Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 17:26:36 +0900
Subject: [PATCH v2 01/11] Refactor some TOAST value ID code to use Oid8
instead of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 14 +++--
contrib/amcheck/verify_heapam.c | 56 +++++++++++--------
6 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..ff234c652230 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..ea4a473ff03c 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1983,7 +1983,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 77d42e7ed65a..11e3d39d1bd4 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);
-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)
+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)
+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..747b8c56fc95 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -725,7 +725,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",
+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -737,13 +737,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -752,7 +752,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",
+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -781,7 +781,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",
+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 6aed63463663..23de9b6c9433 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */
+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -4997,7 +4997,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5021,7 +5021,7 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;
+ Oid8 chunk_id;
int32 chunk_seq;
if (txn->toast_hash == NULL)
@@ -5048,11 +5048,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5161,6 +5161,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5181,13 +5182,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,
+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 20ff58aa7825..b67fe0cc3f37 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1563,6 +1563,9 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
bool isnull;
int32 chunksize;
int32 expected_size;
+ Oid8 toast_valueid;
+
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1570,16 +1573,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",
+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",
+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1590,8 +1593,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",
+ toast_valueid,
chunk_seq));
return;
}
@@ -1610,8 +1613,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",
+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1622,8 +1625,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",
+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1633,8 +1636,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",
+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1665,6 +1668,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external toast_pointer;
@@ -1773,12 +1777,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",
- toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1805,16 +1810,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",
- toast_pointer.va_valueid, cmid));
+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",
+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",
+ toast_pointer_valueid));
return true;
}
@@ -1822,8 +1827,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",
+ toast_pointer_valueid));
return true;
}
@@ -1868,6 +1873,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;
@@ -1898,14 +1904,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;
+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " not found in toast table",
+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",
+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.55.0
v2-0002-Minimize-footprint-of-TOAST_MAX_CHUNK_SIZE-in-hea.patch (text/plain, 4.6K) download
From 6c781b352004591e283dc1abd91026f0a2527a68 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 17:40:13 +0900
Subject: [PATCH v2 02/11] Minimize footprint of TOAST_MAX_CHUNK_SIZE in heap
and amcheck
This eases a follow-up change to support 8-byte TOAST value IDs, as the
maximum chunk size allowed for a single chunk of TOASTed data depends on
the size of the value ID.
---
src/backend/access/heap/heaptoast.c | 20 ++++++++++++--------
contrib/amcheck/verify_heapam.c | 13 +++++++++----
2 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 747b8c56fc95..d1f9c25e93cd 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -634,11 +634,12 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
SysScanDesc toastscan;
HeapTuple ttup;
int32 expectedchunk;
- int32 totalchunks = ((attrsize - 1) / TOAST_MAX_CHUNK_SIZE) + 1;
+ int32 totalchunks;
int startchunk;
int endchunk;
int num_indexes;
int validIndex;
+ int32 max_chunk_size;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -646,8 +647,11 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
- startchunk = sliceoffset / TOAST_MAX_CHUNK_SIZE;
- endchunk = (sliceoffset + slicelength - 1) / TOAST_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+
+ totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
+ startchunk = sliceoffset / max_chunk_size;
+ endchunk = (sliceoffset + slicelength - 1) / max_chunk_size;
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
@@ -747,8 +751,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
- expected_size = curchunk < totalchunks - 1 ? TOAST_MAX_CHUNK_SIZE
- : attrsize - ((totalchunks - 1) * TOAST_MAX_CHUNK_SIZE);
+ expected_size = curchunk < totalchunks - 1 ? max_chunk_size
+ : attrsize - ((totalchunks - 1) * max_chunk_size);
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -763,12 +767,12 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
chcpystrt = 0;
chcpyend = chunksize - 1;
if (curchunk == startchunk)
- chcpystrt = sliceoffset % TOAST_MAX_CHUNK_SIZE;
+ chcpystrt = sliceoffset % max_chunk_size;
if (curchunk == endchunk)
- chcpyend = (sliceoffset + slicelength - 1) % TOAST_MAX_CHUNK_SIZE;
+ chcpyend = (sliceoffset + slicelength - 1) % max_chunk_size;
memcpy(VARDATA(result) +
- curchunk * TOAST_MAX_CHUNK_SIZE - sliceoffset + chcpystrt,
+ (curchunk * max_chunk_size - sliceoffset) + chcpystrt,
chunkdata + chcpystrt,
(chcpyend - chcpystrt) + 1);
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index b67fe0cc3f37..22bba6161de4 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1558,15 +1558,19 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 extsize)
{
int32 chunk_seq;
- int32 last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;
+ int32 last_chunk_seq;
Pointer chunk;
bool isnull;
int32 chunksize;
int32 expected_size;
Oid8 toast_valueid;
+ int32 max_chunk_size;
toast_valueid = ta->toast_pointer.va_valueid;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
+
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
ctx->toast_rel->rd_att, &isnull));
@@ -1631,8 +1635,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
return;
}
- expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
- : extsize - (last_chunk_seq * TOAST_MAX_CHUNK_SIZE);
+ expected_size = chunk_seq < last_chunk_seq ? max_chunk_size
+ : extsize - (last_chunk_seq * max_chunk_size);
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
@@ -1874,9 +1878,10 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
Oid8 toast_valueid;
+ int32 max_chunk_size = TOAST_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / TOAST_MAX_CHUNK_SIZE;
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
--
2.55.0
v2-0003-Rename-varatt_external-to-varatt_external_oid.patch (text/plain, 24.6K) download
From 612f375cddc313a0f31a5a083e05fa56a5702502 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 18:28:10 +0900
Subject: [PATCH v2 03/11] Rename varatt_external to varatt_external_oid
This impacts a few more things, related to this structure:
- VARTAG_ONDISK -> VARTAG_ONDISK_OID
- TOAST_POINTER_SIZE -> TOAST_OID_POINTER_SIZE
- TOAST_MAX_CHUNK_SIZE -> TOAST_OID_MAX_CHUNK_SIZE
- Macros in varatt.h for varatt_external_oid.
This rename is in preparation of a follow-up commit that aims at adding
support for multiple types of external on-disk TOAST pointers, where the
OID type is only one subset of them.
---
src/include/access/detoast.h | 4 +--
src/include/access/heaptoast.h | 6 ++--
src/include/varatt.h | 36 ++++++++++---------
src/backend/access/common/detoast.c | 30 ++++++++--------
src/backend/access/common/toast_compression.c | 6 ++--
src/backend/access/common/toast_internals.c | 16 ++++-----
src/backend/access/heap/heaptoast.c | 2 +-
src/backend/access/table/toast_helper.c | 4 +--
src/backend/access/transam/xlog.c | 8 ++---
.../replication/logical/reorderbuffer.c | 6 ++--
src/backend/utils/adt/varlena.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 2 +-
doc/src/sgml/func/func-info.sgml | 2 +-
doc/src/sgml/storage.sgml | 2 +-
contrib/amcheck/verify_heapam.c | 14 ++++----
15 files changed, 72 insertions(+), 68 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..2b1428b1e81c 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -14,7 +14,7 @@
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
- * into a local "varatt_external" toast pointer. This should be
+ * into a local "varatt_external_oid" toast pointer. This should be
* just a memcpy, but some versions of gcc seem to produce broken code
* that assumes the datum contents are aligned. Introducing an explicit
* intermediate "varattrib_1b_e *" variable seems to fix it.
@@ -28,7 +28,7 @@ do { \
} while (0)
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
-#define TOAST_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external))
+#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index ff234c652230..aaae6e0fef69 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,19 +69,19 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_MAX_CHUNK_SIZE data bytes. This number *must*
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
* be small enough that the completed toast-table tuple (including the
* ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
#define EXTERN_TUPLE_MAX_SIZE MaximumBytesPerTuple(EXTERN_TUPLES_PER_PAGE)
-#define TOAST_MAX_CHUNK_SIZE \
+#define TOAST_OID_MAX_CHUNK_SIZE \
(EXTERN_TUPLE_MAX_SIZE - \
MAXALIGN(SizeofHeapTupleHeader) - \
sizeof(Oid) - \
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 000bdf33b923..33979bbaaad1 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -16,7 +16,7 @@
#define VARATT_H
/*
- * varatt_external is a traditional "TOAST pointer", that is, the
+ * varatt_external_oid is a traditional "TOAST pointer", that is, the
* information needed to fetch a Datum stored out-of-line in a TOAST table.
* The data is compressed if and only if the external size stored in
* va_extinfo is less than va_rawsize - VARHDRSZ.
@@ -29,14 +29,14 @@
* you can look at these fields! (The reason we use memcmp is to avoid
* having to do that just to detect equality of two TOAST pointers...)
*/
-typedef struct varatt_external
+typedef struct varatt_external_oid
{
int32 va_rawsize; /* Original data size (includes header) */
uint32 va_extinfo; /* External saved size (without header) and
* compression method */
Oid va_valueid; /* Unique ID of value within TOAST table */
Oid va_toastrelid; /* RelID of TOAST table containing it */
-} varatt_external;
+} varatt_external_oid;
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
@@ -51,7 +51,7 @@ typedef struct varatt_external
* The creator of such a Datum is entirely responsible that the referenced
* storage survives for as long as referencing pointer Datums can exist.
*
- * Note that just as for varatt_external, this struct is stored
+ * Note that just as for varatt_external_oid, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct varatt_indirect
@@ -66,7 +66,7 @@ typedef struct varatt_indirect
* storage. APIs for this, in particular the definition of struct
* ExpandedObjectHeader, are in src/include/utils/expandeddatum.h.
*
- * Note that just as for varatt_external, this struct is stored
+ * Note that just as for varatt_external_oid, this struct is stored
* unaligned within any containing tuple.
*/
typedef struct ExpandedObjectHeader ExpandedObjectHeader;
@@ -78,15 +78,16 @@ typedef struct varatt_expanded
/*
* Type tag for the various sorts of "TOAST pointer" datums. The peculiar
- * value for VARTAG_ONDISK comes from a requirement for on-disk compatibility
- * with a previous notion that the tag field was the pointer datum's length.
+ * value for VARTAG_ONDISK_OID comes from a requirement for on-disk
+ * compatibility with a previous notion that the tag field was the pointer
+ * datum's length.
*/
typedef enum vartag_external
{
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
- VARTAG_ONDISK = 18
+ VARTAG_ONDISK_OID = 18
} vartag_external;
/* Is a TOAST pointer either type of expanded-object pointer? */
@@ -105,8 +106,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_indirect);
else if (VARTAG_IS_EXPANDED(tag))
return sizeof(varatt_expanded);
- else if (tag == VARTAG_ONDISK)
- return sizeof(varatt_external);
+ else if (tag == VARTAG_ONDISK_OID)
+ return sizeof(varatt_external_oid);
else
{
Assert(false);
@@ -360,7 +361,7 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK;
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
/* Is varlena datum an indirect pointer? */
@@ -502,15 +503,18 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
return ((const varattrib_4b *) PTR)->va_compressed.va_tcinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Same for external Datums; but note argument is a varatt_external */
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid.
+ */
static inline Size
-VARATT_EXTERNAL_GET_EXTSIZE(varatt_external toast_pointer)
+VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer)
{
return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
}
static inline uint32
-VARATT_EXTERNAL_GET_COMPRESS_METHOD(varatt_external toast_pointer)
+VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
{
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
@@ -533,9 +537,9 @@ VARATT_EXTERNAL_GET_COMPRESS_METHOD(varatt_external toast_pointer)
* actually saves space, so we expect either equality or less-than.
*/
static inline bool
-VARATT_EXTERNAL_IS_COMPRESSED(varatt_external toast_pointer)
+VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
{
- return VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) <
+ return VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer) <
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..a50aea7b6b12 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,12 +225,12 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +240,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,7 +251,7 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_GET_COMPRESS_METHOD(toast_pointer) ==
+ if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
@@ -344,7 +344,7 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
int32 attrsize;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
@@ -353,11 +353,11 @@ toast_fetch_datum(varlena *attr)
/* Must copy to access aligned fields */
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- attrsize = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
else
SET_VARSIZE(result, attrsize + VARHDRSZ);
@@ -398,7 +398,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
int32 attrsize;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
@@ -412,9 +412,9 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
+ Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
- attrsize = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
if (sliceoffset >= attrsize)
{
@@ -427,7 +427,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +440,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -550,7 +550,7 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
result = toast_pointer.va_rawsize;
@@ -610,10 +610,10 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 5a5d579494a2..3b1bd6519261 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -262,12 +262,12 @@ toast_get_compression_id(varlena *attr)
*/
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
- cmid = VARATT_EXTERNAL_GET_COMPRESS_METHOD(toast_pointer);
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ cmid = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
}
else if (VARATT_IS_COMPRESSED(attr))
cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 11e3d39d1bd4..ca5ed8dfabca 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -124,7 +124,7 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -176,7 +176,7 @@ toast_save_datum(Relation rel, Datum value,
VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer));
+ Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
}
else
{
@@ -225,7 +225,7 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_valueid = InvalidOid;
if (oldexternal != NULL)
{
- varatt_external old_toast_pointer;
+ varatt_external_oid old_toast_pointer;
Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
/* Must copy to access aligned fields */
@@ -289,7 +289,7 @@ toast_save_datum(Relation rel, Datum value,
{
alignas(int32) varlena hdr;
/* this is to make the union big enough for a chunk: */
- char data[TOAST_MAX_CHUNK_SIZE + VARHDRSZ];
+ char data[TOAST_OID_MAX_CHUNK_SIZE + VARHDRSZ];
} chunk_data;
int32 chunk_size;
@@ -298,7 +298,7 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
/*
* Build a tuple and store it
@@ -359,8 +359,8 @@ toast_save_datum(Relation rel, Datum value,
/*
* Create the TOAST pointer value that we'll return
*/
- result = (varlena *) palloc(TOAST_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK);
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
return PointerGetDatum(result);
@@ -376,7 +376,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index d1f9c25e93cd..81154c17376c 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -647,7 +647,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
- max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..6f2d691cc681 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -171,7 +171,7 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_POINTER_SIZE);
+ * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
* if not, no benefit is to be expected by compressing it.
*
* The return value is the index of the biggest suitable column, or
@@ -184,7 +184,7 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_POINTER_SIZE);
+ int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b23d8bbbdad6..b569690d84dc 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4315,7 +4315,7 @@ WriteControlFile(void)
ControlFile->nameDataLen = NAMEDATALEN;
ControlFile->indexMaxKeys = INDEX_MAX_KEYS;
- ControlFile->toast_max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ ControlFile->toast_max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
ControlFile->loblksize = LOBLKSIZE;
ControlFile->float8ByVal = true; /* vestigial */
@@ -4568,15 +4568,15 @@ ReadControlFile(void)
"INDEX_MAX_KEYS", ControlFile->indexMaxKeys,
"INDEX_MAX_KEYS", INDEX_MAX_KEYS),
errhint("It looks like you need to recompile or initdb.")));
- if (ControlFile->toast_max_chunk_size != TOAST_MAX_CHUNK_SIZE)
+ if (ControlFile->toast_max_chunk_size != TOAST_OID_MAX_CHUNK_SIZE)
ereport(FATAL,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("database files are incompatible with server"),
/* translator: %s is a variable name and %d is its value */
errdetail("The database cluster was initialized with %s %d,"
" but the server was compiled with %s %d.",
- "TOAST_MAX_CHUNK_SIZE", ControlFile->toast_max_chunk_size,
- "TOAST_MAX_CHUNK_SIZE", (int) TOAST_MAX_CHUNK_SIZE),
+ "TOAST_OID_MAX_CHUNK_SIZE", ControlFile->toast_max_chunk_size,
+ "TOAST_OID_MAX_CHUNK_SIZE", (int) TOAST_OID_MAX_CHUNK_SIZE),
errhint("It looks like you need to recompile or initdb.")));
if (ControlFile->loblksize != LOBLKSIZE)
ereport(FATAL,
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 23de9b6c9433..518d1ecad4f2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5155,7 +5155,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *varlena_pointer;
/* va_rawsize is the size of the original datum -- including header */
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
@@ -5225,10 +5225,10 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer));
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
/* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
else
SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0c6d3ba4d22e..aef4f9a536d0 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,7 +4261,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 1542a56ca4b1..79f3085d7696 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -737,7 +737,7 @@ GuessControlValues(void)
ControlFile.xlog_seg_size = DEFAULT_XLOG_SEG_SIZE;
ControlFile.nameDataLen = NAMEDATALEN;
ControlFile.indexMaxKeys = INDEX_MAX_KEYS;
- ControlFile.toast_max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ ControlFile.toast_max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
ControlFile.loblksize = LOBLKSIZE;
ControlFile.float8ByVal = true; /* vestigial */
diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml
index 122fc740f1aa..8f18d3da1e6f 100644
--- a/doc/src/sgml/func/func-info.sgml
+++ b/doc/src/sgml/func/func-info.sgml
@@ -3589,7 +3589,7 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
</row>
<row>
- <entry><structfield>max_toast_chunk_size</structfield></entry>
+ <entry><structfield>max_toast_oid_chunk_size</structfield></entry>
<entry><type>integer</type></entry>
</row>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 19924b98d71d..90aae3defcbc 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,7 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 22bba6161de4..119a54a9404c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,7 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1568,7 +1568,7 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
toast_valueid = ta->toast_pointer.va_valueid;
- max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
/* Sanity-check the sequence number. */
@@ -1674,7 +1674,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
uint16 infomask;
Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
- varatt_external toast_pointer;
+ varatt_external_oid toast_pointer;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1733,7 +1733,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK)
+ if (va_tag != VARTAG_ONDISK_OID)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1791,7 +1791,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
{
ToastCompressionId cmid;
bool valid = false;
@@ -1878,9 +1878,9 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
Oid8 toast_valueid;
- int32 max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
- extsize = VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
--
2.55.0
v2-0004-Switch-pg_column_toast_chunk_id-return-value-from.patch (text/plain, 5.7K) download
From b1db66c1497ce7f92c676ce2776b77db95209fe7 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v2 04/11] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f8a021987b5e..45ed83d28f67 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7849,7 +7849,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',
+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index aef4f9a536d0..6e89e7d596ab 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);
+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out
index cb14ddcee34b..21859fc0dce2 100644
--- a/src/test/isolation/expected/cluster-toast-value-reuse.out
+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out
@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec
index 9a2d10600b39..96cf899db296 100644
--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec
+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec
@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
index fa76bcf28ef9..cc8f034d0167 100644
--- a/src/test/modules/injection_points/specs/repack_toast.spec
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);
+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index c3261bff209f..3d5ea1c256be 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 946ee5726cdd..7ce6ad62f090 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 0eae1c1f6160..31e85d284546 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>
+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.55.0
v2-0005-Add-catcache-support-for-OID8OID.patch (text/plain, 1.3K) download
From 7dc170373ce34699b99752514ed839da5ed153db Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 20:00:30 +0900
Subject: [PATCH v2 05/11] Add catcache support for OID8OID
This is required to be able to do catalog cache lookups of oid8 fields
for toast values of the same type.
---
src/backend/utils/cache/catcache.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 0c8955fc61af..6fa27f4833d4 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -248,6 +248,18 @@ int4hashfast(Datum datum)
return murmurhash32((int32) DatumGetInt32(datum));
}
+static bool
+oid8eqfast(Datum a, Datum b)
+{
+ return DatumGetObjectId8(a) == DatumGetObjectId8(b);
+}
+
+static uint32
+oid8hashfast(Datum datum)
+{
+ return murmurhash64(DatumGetObjectId8(datum));
+}
+
static bool
texteqfast(Datum a, Datum b)
{
@@ -311,6 +323,11 @@ GetCCHashEqFuncs(Oid keytype, CCHashFN *hashfunc, RegProcedure *eqfunc, CCFastEq
*fasteqfunc = int4eqfast;
*eqfunc = F_INT4EQ;
break;
+ case OID8OID:
+ *hashfunc = oid8hashfast;
+ *fasteqfunc = oid8eqfast;
+ *eqfunc = F_OID8EQ;
+ break;
case TEXTOID:
*hashfunc = texthashfast;
*fasteqfunc = texteqfast;
--
2.55.0
v2-0006-Add-support-for-TOAST-chunk_id-type-in-binary-upg.patch (text/plain, 9.3K) download
From 5c97a4095a679bc4e0bbd058c11ebefb192bf028 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 10:57:59 +0900
Subject: [PATCH v2 06/11] Add support for TOAST chunk_id type in binary
upgrades
This commit adds a new function, which would set the type of a chunk_id
attribute for a TOAST table across upgrades. This piece currently works
only with chunk_id = OIDOID, but it is required in a follow-up patch
where support for chunk_id = OID8OID is supported on top of the existing
one.
---
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 ++++
src/backend/catalog/heap.c | 1 +
src/backend/catalog/toasting.c | 20 ++++++++++++++++++-
src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++++
src/bin/pg_dump/pg_dump.c | 10 +++++++++-
.../expected/spgist_name_ops.out | 6 ++++--
7 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae443859..117f3e3f7746 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -29,6 +29,7 @@ extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_index_pg_class_relfilenumber;
extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber;
+extern PGDLLIMPORT Oid binary_upgrade_next_toast_chunk_id_typoid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 45ed83d28f67..7ad1d4219f99 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11934,6 +11934,10 @@
proname => 'binary_upgrade_set_next_toast_pg_class_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_toast_pg_class_oid' },
+{ oid => '8219', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_toast_chunk_id_typoid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_toast_chunk_id_typoid' },
{ oid => '3589', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_set_next_pg_enum_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 88087654de9b..1539edbe9e9f 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -80,6 +80,7 @@
/* Potentially set by pg_upgrade_support functions */
Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid;
Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid;
+Oid binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
RelFileNumber binary_upgrade_next_heap_pg_class_relfilenumber = InvalidRelFileNumber;
RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber = InvalidRelFileNumber;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 4aa52a4bd253..bc0a72268335 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -146,6 +146,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_chunkid_typid = OIDOID;
/*
* Is it already toasted?
@@ -184,6 +185,23 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!OidIsValid(binary_upgrade_next_toast_pg_class_oid))
return false;
+
+ /*
+ * The attribute type for chunk_id should have been set when requesting
+ * a TOAST table creation.
+ */
+ if (!OidIsValid(binary_upgrade_next_toast_chunk_id_typoid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("toast chunk_id type not set while in binary upgrade mode")));
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
+ binary_upgrade_next_toast_chunk_id_typoid)));
+
+ toast_chunkid_typid = binary_upgrade_next_toast_chunk_id_typoid;
+ binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
}
/*
@@ -205,7 +223,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,
+ toast_chunkid_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4feeb..06b789d71ded 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -149,6 +149,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_toast_chunk_id_typoid(PG_FUNCTION_ARGS)
+{
+ Oid typoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_toast_chunk_id_typoid = typoid;
+
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4948e6d80c7e..b7d31e0aed3f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -104,6 +104,7 @@ typedef struct
RelFileNumber relfilenumber; /* object filenode */
Oid toast_oid; /* toast table OID */
RelFileNumber toast_relfilenumber; /* toast table filenode */
+ Oid toast_chunk_id_typoid; /* type of chunk_id attribute */
Oid toast_index_oid; /* toast table index OID */
RelFileNumber toast_index_relfilenumber; /* toast table index filenode */
} BinaryUpgradeClassOidItem;
@@ -5872,7 +5873,10 @@ collectBinaryUpgradeClassOids(Archive *fout)
const char *query;
query = "SELECT c.oid, c.relkind, c.relfilenode, c.reltoastrelid, "
- "ct.relfilenode, i.indexrelid, cti.relfilenode "
+ "ct.relfilenode, i.indexrelid, cti.relfilenode, "
+ "(SELECT a.atttypid FROM pg_attribute AS a "
+ " WHERE a.attrelid = c.reltoastrelid AND attname = 'chunk_id'::text) "
+ " AS toastchunktypid "
"FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_index i "
"ON (c.reltoastrelid = i.indrelid AND i.indisvalid) "
"LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) "
@@ -5894,6 +5898,7 @@ collectBinaryUpgradeClassOids(Archive *fout)
binaryUpgradeClassOids[i].toast_relfilenumber = atooid(PQgetvalue(res, i, 4));
binaryUpgradeClassOids[i].toast_index_oid = atooid(PQgetvalue(res, i, 5));
binaryUpgradeClassOids[i].toast_index_relfilenumber = atooid(PQgetvalue(res, i, 6));
+ binaryUpgradeClassOids[i].toast_chunk_id_typoid = atooid(PQgetvalue(res, i, 7));
}
PQclear(res);
@@ -5958,6 +5963,9 @@ binary_upgrade_set_pg_class_oids(Archive *fout,
appendPQExpBuffer(upgrade_buffer,
"SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n",
entry->toast_relfilenumber);
+ appendPQExpBuffer(upgrade_buffer,
+ "SELECT pg_catalog.binary_upgrade_set_next_toast_chunk_id_typoid('%u'::pg_catalog.oid);\n",
+ entry->toast_chunk_id_typoid);
/* every toast table has an index */
appendPQExpBuffer(upgrade_buffer,
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede2430..35e59d0cd83c 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -61,9 +61,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -110,9 +111,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.55.0
v2-0007-Enlarge-OID-generation-to-8-bytes.patch (text/plain, 14.5K) download
From b68f2c8b2c5de6f68045093ce8bbb489461a49de Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:15:15 +0900
Subject: [PATCH v2 07/11] Enlarge OID generation to 8 bytes
This adds a new routine called GetNewObjectId8() in varsup.c, which is
able to retrieve a 64b OID. GetNewObjectId() is kept compatible with
its origin, where we still check that the lower 32 bits of the counter
do not wraparound, handling the FirstNormalObjectId case.
pg_resetwal -o/--next-oid is updated to be able to handle 8-byte OIDs.
---
src/include/access/transam.h | 3 +-
src/include/access/xlog.h | 2 +-
src/include/catalog/pg_control.h | 2 +-
src/backend/access/rmgrdesc/xlogdesc.c | 8 +--
src/backend/access/transam/varsup.c | 62 ++++++++++++++++-------
src/backend/access/transam/xlog.c | 8 +--
src/backend/access/transam/xlogrecovery.c | 2 +-
src/bin/pg_controldata/pg_controldata.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 10 ++--
doc/src/sgml/ref/pg_resetwal.sgml | 6 +--
10 files changed, 66 insertions(+), 39 deletions(-)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 55a4ab26b348..f687df1edc60 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -211,7 +211,7 @@ typedef struct TransamVariablesData
/*
* These fields are protected by OidGenLock.
*/
- Oid nextOid; /* next OID to assign */
+ Oid8 nextOid; /* next OID (8 bytes) to assign */
uint32 oidCount; /* OIDs available before must do XLOG work */
/*
@@ -353,6 +353,7 @@ extern void SetTransactionIdLimit(TransactionId oldest_datfrozenxid,
extern void AdvanceOldestClogXid(TransactionId oldest_datfrozenxid);
extern bool ForceTransactionIdLimitUpdate(void);
extern Oid GetNewObjectId(void);
+extern Oid8 GetNewObjectId8(void);
extern void StopGeneratingPinnedObjectIds(void);
#ifdef USE_ASSERT_CHECKING
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 4dd986242046..64763ffffc92 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -270,7 +270,7 @@ extern void ShutdownXLOG(int code, Datum arg);
extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
-extern void XLogPutNextOid(Oid nextOid);
+extern void XLogPutNextOid(Oid8 nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern XLogRecPtr XLogAssignLSN(void);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 80b3a730e035..7f708d53ea94 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -43,7 +43,7 @@ typedef struct CheckPoint
int wal_level; /* current wal_level */
bool logicalDecodingEnabled; /* current logical decoding status */
FullTransactionId nextXid; /* next free transaction ID */
- Oid nextOid; /* next free OID */
+ Oid8 nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 2468a7d25785..64e749c1e6b7 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -101,7 +101,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
CheckPoint *checkpoint = (CheckPoint *) rec;
appendStringInfo(buf, "redo %X/%08X; "
- "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "
+ "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid " OID8_FORMAT "; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
@@ -129,10 +129,10 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
else if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
- memcpy(&nextOid, rec, sizeof(Oid));
- appendStringInfo(buf, "%u", nextOid);
+ memcpy(&nextOid, rec, sizeof(Oid8));
+ appendStringInfo(buf, OID8_FORMAT, nextOid);
}
else if (info == XLOG_RESTORE_POINT)
{
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index cb68d8fa974a..03209bc18c18 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -541,31 +541,51 @@ ForceTransactionIdLimitUpdate(void)
/*
- * GetNewObjectId -- allocate a new OID
+ * GetNewObjectId -- allocate a new OID (4 bytes)
*
- * OIDs are generated by a cluster-wide counter. Since they are only 32 bits
- * wide, counter wraparound will occur eventually, and therefore it is unwise
- * to assume they are unique unless precautions are taken to make them so.
- * Hence, this routine should generally not be used directly. The only direct
- * callers should be GetNewOidWithIndex() and GetNewRelFileNumber() in
- * catalog/catalog.c.
+ * OIDs are generated by a cluster-wide counter. The callers of this routine
+ * expect a 32 bit-wide counter, and counter wraparound will occur eventually,
+ * and therefore it is unwise to assume they are unique unless precautions are
+ * taken to make them so. This routine should generally not be used directly.
+ * The only direct callers should be GetNewOidWithIndex() and
+ * GetNewRelFileNumber() in catalog/catalog.c.
*/
Oid
GetNewObjectId(void)
{
- Oid result;
+ return (Oid) GetNewObjectId8();
+}
+
+/*
+ * GetNewObjectId8 -- allocate a new OID (8 bytes)
+ *
+ * This routine can be called directly if the consumer of the OID allocated
+ * stores the counter in an 8-byte space, where wraparound does not matter.
+ * We still need to care about the wraparound case in the low 32 bits of the
+ * space allocated, GetNewObjectId() expecting OIDs to never be allocated
+ * up to FirstNormalObjectId.
+ */
+Oid8
+GetNewObjectId8(void)
+{
+ Oid8 result;
+ Oid nextoid_lo;
+ uint32 nextoid_hi;
/* safety check, we should never get this far in a HS standby */
if (RecoveryInProgress())
elog(ERROR, "cannot assign OIDs during recovery");
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
+ nextoid_lo = (Oid) TransamVariables->nextOid;
+ nextoid_hi = (uint32) (TransamVariables->nextOid >> 32);
/*
- * Check for wraparound of the OID counter. We *must* not return 0
- * (InvalidOid), and in normal operation we mustn't return anything below
- * FirstNormalObjectId since that range is reserved for initdb (see
- * IsCatalogRelationOid()). Note we are relying on unsigned comparison.
+ * Check for wraparound of the OID counter in its lower 4 bytes. We
+ * *must* not return 0 (InvalidOid), and in normal operation we
+ * mustn't return anything below FirstNormalObjectId since that range
+ * is reserved for initdb (see IsCatalogRelationOid()). Note we are
+ * relying on unsigned comparison.
*
* During initdb, we start the OID generator at FirstGenbkiObjectId, so we
* only wrap if before that point when in bootstrap or standalone mode.
@@ -575,26 +595,32 @@ GetNewObjectId(void)
* available for automatic assignment during initdb, while ensuring they
* will never conflict with user-assigned OIDs.
*/
- if (TransamVariables->nextOid < ((Oid) FirstNormalObjectId))
+ if (nextoid_lo < ((Oid) FirstNormalObjectId))
{
if (IsPostmasterEnvironment)
{
/* wraparound, or first post-initdb assignment, in normal mode */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
else
{
/* we may be bootstrapping, so don't enforce the full range */
- if (TransamVariables->nextOid < ((Oid) FirstGenbkiObjectId))
+ if (nextoid_lo < ((Oid) FirstGenbkiObjectId))
{
/* wraparound in standalone mode (unlikely but possible) */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
}
}
+ /*
+ * Set next OID in its 8-byte space, skipping the first post-init
+ * assignment.
+ */
+ TransamVariables->nextOid = ((Oid8) nextoid_hi) << 32 | nextoid_lo;
+
/* If we run out of logged for use oids then we must log more */
if (TransamVariables->oidCount == 0)
{
@@ -619,7 +645,7 @@ GetNewObjectId(void)
* to the specified value.
*/
static void
-SetNextObjectId(Oid nextOid)
+SetNextObjectId(Oid8 nextOid)
{
/* Safety check, this is only allowable during initdb */
if (IsPostmasterEnvironment)
@@ -629,7 +655,7 @@ SetNextObjectId(Oid nextOid)
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
if (TransamVariables->nextOid > nextOid)
- elog(ERROR, "too late to advance OID counter to %u, it is now %u",
+ elog(ERROR, "too late to advance OID counter to " OID8_FORMAT ", it is now " OID8_FORMAT,
nextOid, TransamVariables->nextOid);
TransamVariables->nextOid = nextOid;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b569690d84dc..17d7c969ba22 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8574,10 +8574,10 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
* Write a NEXTOID log record
*/
void
-XLogPutNextOid(Oid nextOid)
+XLogPutNextOid(Oid8 nextOid)
{
XLogBeginInsert();
- XLogRegisterData(&nextOid, sizeof(Oid));
+ XLogRegisterData(&nextOid, sizeof(Oid8));
(void) XLogInsert(RM_XLOG_ID, XLOG_NEXTOID);
/*
@@ -8842,7 +8842,7 @@ xlog_redo(XLogReaderState *record)
if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
/*
* We used to try to take the maximum of TransamVariables->nextOid and
@@ -8851,7 +8851,7 @@ xlog_redo(XLogReaderState *record)
* anyway, better to just believe the record exactly. We still take
* OidGenLock while setting the variable, just in case.
*/
- memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid));
+ memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid8));
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
TransamVariables->nextOid = nextOid;
TransamVariables->oidCount = 0;
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 5f3b065b8942..04eff92e8536 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -831,7 +831,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
LSN_FORMAT_ARGS(checkPoint.redo),
wasShutdown ? "true" : "false"));
ereport(DEBUG1,
- (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: %u",
+ (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: " OID8_FORMAT,
U64FromFullTransactionId(checkPoint.nextXid),
checkPoint.nextOid)));
ereport(DEBUG1,
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 6fc87ed114d8..b785f7f40701 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -269,7 +269,7 @@ main(int argc, char *argv[])
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile->checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile->checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 79f3085d7696..68a176a2c427 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -82,7 +82,7 @@ static TransactionId oldest_commit_ts_xid_val;
static TransactionId newest_commit_ts_xid_val;
static bool next_oid_given = false;
-static Oid next_oid_val;
+static Oid8 next_oid_val;
static bool mxids_given = false;
static MultiXactId next_mxid_val;
@@ -253,7 +253,7 @@ main(int argc, char *argv[])
case 'o':
errno = 0;
- next_oid_val = strtouint32_strict(optarg, &endptr, 0);
+ next_oid_val = strtou64(optarg, &endptr, 0);
if (endptr == optarg || *endptr != '\0' || errno != 0)
{
pg_log_error("invalid argument for option %s", "-o");
@@ -775,7 +775,7 @@ PrintControlValues(bool guessed)
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile.checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile.checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile.checkPointCopy.nextMulti);
@@ -861,7 +861,7 @@ PrintNewControlValues(void)
if (next_oid_given)
{
- printf(_("NextOID: %u\n"),
+ printf(_("NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
}
@@ -1236,7 +1236,7 @@ usage(void)
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
- printf(_(" -o, --next-oid=OID set next OID\n"));
+ printf(_(" -o, --next-oid=OID8 set next OID (8 bytes)\n"));
printf(_(" -O, --multixact-offset=OFFSET set next multitransaction offset\n"));
printf(_(" -u, --oldest-transaction-id=XID set oldest transaction ID\n"));
printf(_(" -x, --next-transaction-id=XID set next transaction ID\n"));
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index 664c525f481c..a9580f35de95 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -286,11 +286,11 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>-o <replaceable class="parameter">oid</replaceable></option></term>
- <term><option>--next-oid=<replaceable class="parameter">oid</replaceable></option></term>
+ <term><option>-o <replaceable class="parameter">oid8</replaceable></option></term>
+ <term><option>--next-oid=<replaceable class="parameter">oid8</replaceable></option></term>
<listitem>
<para>
- Manually set the next OID.
+ Manually set the next OID (8 bytes).
</para>
<para>
--
2.55.0
v2-0008-Add-relation-option-toast_value_type.patch (text/plain, 6.5K) download
From 09a5d38659a41d2a5931401d07a655b08911c963 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:54:58 +0900
Subject: [PATCH v2 08/11] Add relation option toast_value_type
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is created
initially. This parameter has no effect if a TOAST table exists, even
after it is modified later on, even on rewrites.
This can be set only to "oid" currently, and will be expanded with a
second mode later.
Note: perhaps it would make sense to introduce that only when support
for 8-byte OID values are added to TOAST, the split is here to ease
review.
---
src/include/utils/rel.h | 17 +++++++++++++++++
src/backend/access/common/reloptions.c | 21 +++++++++++++++++++++
src/backend/catalog/toasting.c | 6 ++++++
src/bin/psql/tab-complete.in.c | 1 +
doc/src/sgml/ref/create_table.sgml | 18 ++++++++++++++++++
5 files changed, 63 insertions(+)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 89c159b133fa..4d623d7607e4 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -340,11 +340,20 @@ typedef enum StdRdOptIndexCleanup
STDRD_OPTION_VACUUM_INDEX_CLEANUP_ON,
} StdRdOptIndexCleanup;
+/* StdRdOptions->toast_value_type values */
+typedef enum StdRdOptToastValueType
+{
+ STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+} StdRdOptToastValueType;
+
typedef struct StdRdOptions
{
int32 vl_len_; /* varlena header (do not touch directly!) */
int fillfactor; /* page fill factor in percent (0..100) */
int toast_tuple_target; /* target for tuple toasting */
+ StdRdOptToastValueType toast_value_type; /* type assigned to chunk_id
+ * at toast table creation */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
int parallel_workers; /* max number of parallel workers */
@@ -369,6 +378,14 @@ typedef struct StdRdOptions
((relation)->rd_options ? \
((StdRdOptions *) (relation)->rd_options)->toast_tuple_target : (defaulttarg))
+/*
+ * RelationGetToastValueType
+ * Returns the relation's toast_value_type. Note multiple eval of argument!
+ */
+#define RelationGetToastValueType(relation, defaulttarg) \
+ ((relation)->rd_options ? \
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)
+
/*
* RelationGetFillFactor
* Returns the relation's fillfactor. Note multiple eval of argument!
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 3e832c3797e8..66fd2a05538e 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -548,6 +548,14 @@ static relopt_enum_elt_def viewCheckOptValues[] =
{(const char *) NULL} /* list terminator */
};
+/* values from StdRdOptToastValueType */
+static relopt_enum_elt_def StdRdOptToastValueTypes[] =
+{
+ /* no value for INVALID */
+ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {(const char *) NULL} /* list terminator */
+};
+
static relopt_enum enumRelOpts[] =
{
{
@@ -561,6 +569,17 @@ static relopt_enum enumRelOpts[] =
STDRD_OPTION_VACUUM_INDEX_CLEANUP_AUTO,
gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")
},
+ {
+ {
+ "toast_value_type",
+ "Controls the attribute type of chunk_id at toast table creation",
+ RELOPT_KIND_HEAP,
+ ShareUpdateExclusiveLock
+ },
+ StdRdOptToastValueTypes,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ gettext_noop("Valid values are \"oid\".")
+ },
{
{
"buffering",
@@ -2008,6 +2027,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind)
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_analyze_min_duration)},
{"toast_tuple_target", RELOPT_TYPE_INT,
offsetof(StdRdOptions, toast_tuple_target)},
+ {"toast_value_type", RELOPT_TYPE_ENUM,
+ offsetof(StdRdOptions, toast_value_type)},
{"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL,
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)},
{"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL,
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index bc0a72268335..28ddbd10d357 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -159,9 +159,15 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!IsBinaryUpgrade)
{
+ StdRdOptToastValueType value_type;
+
/* Normal mode, normal check */
if (!needs_toast_table(rel))
return false;
+
+ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
+ if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
+ toast_chunkid_typid = OIDOID;
}
else
{
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 190fff7ea0e0..30544cf8ad46 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -1450,6 +1450,7 @@ static const char *const table_storage_parameters[] = {
"toast.vacuum_max_eager_freeze_failure_rate",
"toast.vacuum_truncate",
"toast_tuple_target",
+ "toast_value_type",
"user_catalog_table",
"vacuum_index_cleanup",
"vacuum_max_eager_freeze_failure_rate",
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index e342585c7f08..3da754045c04 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1653,6 +1653,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry id="reloption-toast-value-type" xreflabel="toast_value_type">
+ <term><literal>toast_value_type</literal> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>toast_value_type</varname> storage parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ The toast_value_type specifies the attribute type of
+ <literal>chunk_id</literal> used when initially creating a toast
+ relation for this table.
+ By default this parameter is <literal>oid</literal>, to assign
+ <type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter cannot be set for TOAST tables.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="reloption-parallel-workers" xreflabel="parallel_workers">
<term><literal>parallel_workers</literal> (<type>integer</type>)
<indexterm>
--
2.55.0
v2-0009-Add-support-for-oid8-TOAST-values.patch (text/plain, 17.1K) download
From 42fac593928ca5e341eb1738d51967c6117eaa09 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 6 Aug 2026 13:08:37 +0900
Subject: [PATCH v2 09/11] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, casted to OID (for now).
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 1 +
src/backend/access/common/toast_internals.c | 94 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 19 ++++-
src/backend/catalog/toasting.c | 24 +++++-
doc/src/sgml/ref/create_table.sgml | 2 +
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 22 +++--
9 files changed, 132 insertions(+), 41 deletions(-)
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 4d623d7607e4..f9c59db6ffd9 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -345,6 +345,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 66fd2a05538e..fda776215cdd 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..2c180491c2ff 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,8 +132,10 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -200,24 +203,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OIDOID)
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ Assert(false);
}
else
{
@@ -263,17 +274,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OIDOID)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
}
}
@@ -303,7 +319,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OIDOID)
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ else if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -384,6 +403,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -395,6 +415,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -405,10 +427,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +485,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +493,24 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..a93e98788da7 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +658,18 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 28ddbd10d357..66d483bb1be3 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -168,6 +169,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
toast_chunkid_typid = OIDOID;
+ else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)
+ toast_chunkid_typid = OID8OID;
}
else
{
@@ -200,7 +203,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -225,6 +229,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -343,7 +360,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OIDOID)
+ opclassIds[0] = OID_BTREE_OPS_OID;
+ else if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3da754045c04..3ed8f3581177 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1666,6 +1666,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
relation for this table.
By default this parameter is <literal>oid</literal>, to assign
<type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter can be set to <type>oid8</type> to use <type>oid8</type>
+ as attribute type for <literal>chunk_id</literal>.
This parameter cannot be set for TOAST tables.
</para>
</listitem>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 90aae3defcbc..21bd67c11ea0 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an OID8 identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 119a54a9404c..ad6c37eefae6 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1878,7 +1878,11 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
Oid8 toast_valueid;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1886,10 +1890,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v2-0010-Add-tests-for-TOAST-relations-with-oid8-as-value-.patch (text/plain, 21.5K) download
From 3989018aadd31299407ddf6a0eff56ed25bc3b06 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 8 May 2026 10:20:36 +0900
Subject: [PATCH v2 10/11] Add tests for TOAST relations with oid8 as value
type
This adds coverage for relations created with default_toast_type =
'oid8', for external TOAST pointers both compressed and uncompressed.
---
src/test/regress/expected/strings.out | 231 ++++++++++++++++++----
src/test/regress/expected/type_sanity.out | 6 +-
src/test/regress/sql/strings.sql | 134 +++++++++----
src/test/regress/sql/type_sanity.sql | 6 +-
4 files changed, 296 insertions(+), 81 deletions(-)
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 313c50394650..af95a4725d08 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,72 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
substr
--------
567890
@@ -2105,7 +2263,6 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index 1d21d3eb4467..4fabe08f74c0 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -578,15 +578,15 @@ WHERE c1.relnatts != (SELECT count(*) FROM pg_attribute AS a1
(0 rows)
-- Cross-check against pg_type entry
--- NOTE: we allow attstorage to be 'plain' even when typstorage is not;
--- this is mainly for toast tables.
+-- NOTE: we allow attstorage to be 'plain' or 'external' even when typstorage
+-- is not; this is mainly for toast tables.
SELECT a1.attrelid, a1.attname, t1.oid, t1.typname
FROM pg_attribute AS a1, pg_type AS t1
WHERE a1.atttypid = t1.oid AND
(a1.attlen != t1.typlen OR
a1.attalign != t1.typalign OR
a1.attbyval != t1.typbyval OR
- (a1.attstorage != t1.typstorage AND a1.attstorage != 'p'));
+ (a1.attstorage != t1.typstorage AND a1.attstorage NOT IN ('e', 'p')));
attrelid | attname | oid | typname
----------+---------+-----+---------
(0 rows)
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 38946e8954df..cc16bb090199 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,147 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-- test internally compressing datums
diff --git a/src/test/regress/sql/type_sanity.sql b/src/test/regress/sql/type_sanity.sql
index 95d5b6e09151..5c166f00bb7e 100644
--- a/src/test/regress/sql/type_sanity.sql
+++ b/src/test/regress/sql/type_sanity.sql
@@ -420,8 +420,8 @@ WHERE c1.relnatts != (SELECT count(*) FROM pg_attribute AS a1
WHERE a1.attrelid = c1.oid AND a1.attnum > 0);
-- Cross-check against pg_type entry
--- NOTE: we allow attstorage to be 'plain' even when typstorage is not;
--- this is mainly for toast tables.
+-- NOTE: we allow attstorage to be 'plain' or 'external' even when typstorage
+-- is not; this is mainly for toast tables.
SELECT a1.attrelid, a1.attname, t1.oid, t1.typname
FROM pg_attribute AS a1, pg_type AS t1
@@ -429,7 +429,7 @@ WHERE a1.atttypid = t1.oid AND
(a1.attlen != t1.typlen OR
a1.attalign != t1.typalign OR
a1.attbyval != t1.typbyval OR
- (a1.attstorage != t1.typstorage AND a1.attstorage != 'p'));
+ (a1.attstorage != t1.typstorage AND a1.attstorage NOT IN ('e', 'p')));
-- Look for IsCatalogTextUniqueIndexOid() omissions.
--
2.55.0
v2-0011-Add-support-for-64-bit-TOAST-pointers.patch (text/plain, 43.3K) download
From aaba8c09fe75b24cbd08631fffcc0ada2341ff3e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 6 Aug 2026 13:21:21 +0900
Subject: [PATCH v2 11/11] Add support for 64-bit TOAST pointers
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 3 +
src/include/access/heaptoast.h | 16 +-
src/include/varatt.h | 78 ++++-
src/backend/access/common/detoast.c | 171 +++++++---
src/backend/access/common/reloptions.c | 2 +-
src/backend/access/common/toast_internals.c | 314 ++++++++++++------
src/backend/access/heap/heaptoast.c | 5 +-
.../replication/logical/reorderbuffer.c | 44 ++-
doc/src/sgml/storage.sgml | 10 +-
contrib/amcheck/verify_heapam.c | 102 ++++--
10 files changed, 536 insertions(+), 209 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..f065ea034e98 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -30,6 +30,9 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..b7ef69a8f56b 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,13 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 33979bbaaad1..6d7ed246fe2f 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -38,6 +38,34 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except that va_valueid is split into two uint32 halves to enforce a
+ * stricter structure size without padding.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+/* Helper macros to get/set the 64-bit value ID from the lo/hi fields */
+#define VARATT_EXTERNAL_OID8_GET_VALUEID(tp) \
+ ((Oid8) (tp).va_valueid_lo | ((Oid8) (tp).va_valueid_hi << 32))
+#define VARATT_EXTERNAL_OID8_SET_VALUEID(tp, id) \
+ do { \
+ (tp).va_valueid_lo = (uint32) (id); \
+ (tp).va_valueid_hi = (uint32) ((id) >> 32); \
+ } while (0)
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -87,6 +115,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -108,6 +137,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -361,7 +392,19 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
+}
+
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
+{
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
}
/* Is varlena datum an indirect pointer? */
@@ -519,16 +562,6 @@ VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
* Testing whether an externally-stored value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -543,4 +576,27 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid8.
+ */
+static inline Size
+VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+}
+
+static inline uint32
+VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+}
+
+static inline bool
+VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)
+{
+ return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <
+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+}
+
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..e1919809751c 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,40 +225,51 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /* Extract extsize and compression info from the appropriate pointer */
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
- * For compressed values, we need to fetch enough slices to decompress
- * at least the requested part (when a prefix is requested).
- * Otherwise, just fetch all slices.
+ * For compressed values, we need to fetch enough slices to
+ * decompress at least the requested part (when a prefix is
+ * requested). Otherwise, just fetch all slices.
+ *
+ * At least for now, if it's LZ4 data, we'll have to fetch the
+ * whole thing, because there doesn't seem to be an API call to
+ * determine how much compressed data we need to be sure of being
+ * able to decompress the required slice.
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
- /*
- * Determine maximum amount of compressed data needed for a prefix
- * of a given length (after decompression).
- *
- * At least for now, if it's LZ4 data, we'll have to fetch the
- * whole thing, because there doesn't seem to be an API call to
- * determine how much compressed data we need to be sure of being
- * able to decompress the required slice.
- */
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,23 +355,47 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- result = (varlena *) palloc(attrsize + VARHDRSZ);
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
+
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -369,10 +404,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +433,43 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +482,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +495,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +504,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +605,20 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +675,20 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index fda776215cdd..7d77764af1a8 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -579,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")
+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 2c180491c2ff..7b37798e1c08 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -125,7 +125,6 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -133,9 +132,17 @@ toast_save_datum(Relation rel, Datum value,
int num_indexes;
int validIndex;
Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
Assert(OidIsValid(toast_typid));
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -165,28 +172,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -198,13 +209,14 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or OID8 TOAST relations.
+ * Choose a new value to use as the value ID for this toast value, and
+ * determine the maximum chunk size based on the TOAST table's chunk_id
+ * type.
*
* Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
@@ -213,84 +225,123 @@ toast_save_datum(Relation rel, Datum value,
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * conflict with either new or existing toast value IDs. For Oid8 tables,
+ * value conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OIDOID)
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
+ if (!OidIsValid(rel->rd_toastoid))
+ {
+ /* normal case: just choose a new Oid8 */
+ va_valueid = GetNewObjectId8();
+ }
else
- Assert(false);
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid8;
+ if (oldexternal != NULL)
+ {
+ varatt_external_oid8 old_toast_pointer;
+
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /* This value came from the old toast table; reuse its ID */
+ va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);
+ /*
+ * There is a corner case here: the table rewrite might
+ * have to copy both live and recently-dead versions of a
+ * row, and those versions could easily reference the same
+ * toast value. When we copy the second or later version
+ * of such a row, reusing the OID will mean we select an
+ * OID that's already in the new toast table. Check for
+ * that, and if so, just fall through without writing the
+ * data again.
+ *
+ * While annoying and ugly-looking, this is a good thing
+ * because it ensures that we wind up with only one copy
+ * of the toast value when there is only one copy in the
+ * old toast table. Before we detected this case, we'd
+ * have made multiple copies, wasting space; and what's
+ * worse, the copies belonging to already-deleted heap
+ * tuples would not be reclaimed by VACUUM.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
+ }
+ }
+ if (va_valueid == InvalidOid8)
+ {
+ /*
+ * New value. For Oid8 we just pick a new ID without worrying
+ * about conflicts.
+ */
+ va_valueid = GetNewObjectId8();
+ }
+ }
+
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
+ if (!OidIsValid(rel->rd_toastoid))
{
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ /* normal case: just choose an unused OID */
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
+ else
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid;
+ if (oldexternal != NULL)
{
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
+ varatt_external_oid old_toast_pointer;
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
{
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
+ /* This value came from the old toast table; reuse its OID */
+ va_valueid = old_toast_pointer.va_valueid;
+
+ /*
+ * Corner case during table rewrite with multiple
+ * versions of the same row. See above for details.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
}
}
- }
- if (toast_pointer.va_valueid == InvalidOid)
- {
- /*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
- */
- if (toast_typid == OIDOID)
+ if (va_valueid == InvalidOid)
{
+ /*
+ * new value; must choose a value that doesn't conflict in
+ * either old or new toast table.
+ */
do
{
- toast_pointer.va_valueid =
+ va_valueid =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ va_valueid));
}
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
}
+
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
/*
@@ -314,15 +365,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
- if (toast_typid == OIDOID)
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
- else if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -376,11 +427,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -395,7 +469,6 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -403,42 +476,75 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toast_typid;
+ Oid toastrelid;
+ vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
/*
- * Open the toast relation and its indexes
+ * Determine the pointer type from the datum's vartag and extract the
+ * toast relation OID and value ID accordingly. The vartag tells us
+ * everything we need - no TOAST table schema lookup required.
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
- toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ tag = VARTAG_EXTERNAL(attr);
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
+ if (tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- if (toast_typid == OIDOID)
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toastrelid = toast_pointer8.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ Assert(tag == VARTAG_ONDISK_OID);
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toastrelid = toast_pointer.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(toast_pointer.va_valueid));
- else
- Assert(false);
+ }
/*
* Find all the chunks. (We don't actually care whether we see them in
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index a93e98788da7..364aec11fc88 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -650,7 +650,10 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ if (toast_typid == OID8OID)
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+ else
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 518d1ecad4f2..b87939bc0867 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5156,12 +5156,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* va_rawsize is the size of the original datum -- including header */
varatt_external_oid toast_pointer;
+ varatt_external_oid8 toast_pointer8;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
+ int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5181,8 +5183,19 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (!VARATT_IS_EXTERNAL(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ /* Branch on vartag to handle both pointer types */
+ if (VARTAG_EXTERNAL(varlena_pointer) == VARTAG_ONDISK_OID8)
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);
+ toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ rawsize = toast_pointer8.va_rawsize;
+ }
+ else
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
+ rawsize = toast_pointer.va_rawsize;
+ }
/*
* Check whether the toast tuple changed, replace if so.
@@ -5200,7 +5213,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(rawsize);
ent->reconstructed = reconstructed;
@@ -5225,13 +5238,28 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ /* Verify size and set compression status based on pointer type */
+ if (VARTAG_EXTERNAL(varlena_pointer) == VARTAG_ONDISK_OID8)
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 21bd67c11ea0..3561a6a67ece 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,9 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the
+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index ad6c37eefae6..0e1c83958af3 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1566,9 +1568,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
Oid8 toast_valueid;
int32 max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
+ toast_valueid = ta->va_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
/* Sanity-check the sequence number. */
@@ -1673,8 +1677,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1733,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1778,26 +1784,46 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
/*
- * Must copy attr into toast_pointer for alignment considerations
+ * Must copy attr into toast_pointer for alignment considerations.
+ * Branch on the tag to determine which pointer type to extract.
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ va_tag_value = VARTAG_EXTERNAL(attr);
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ va_rawsize = toast_pointer8.va_rawsize;
+ va_extinfo = toast_pointer8.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
+ va_rawsize = toast_pointer.va_rawsize;
+ va_extinfo = toast_pointer.va_extinfo;
+ }
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1851,7 +1877,23 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ ta->tag = va_tag_value;
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 tp;
+
+ VARATT_EXTERNAL_GET_POINTER(tp, attr);
+ ta->va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(tp);
+ ta->va_extinfo = tp.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid tp;
+
+ VARATT_EXTERNAL_GET_POINTER(tp, attr);
+ ta->va_valueid = tp.va_valueid;
+ ta->va_extinfo = tp.va_extinfo;
+ }
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1879,29 +1921,33 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
Oid8 toast_valueid;
int32 max_chunk_size;
- Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
+ if (ta->tag == VARTAG_ONDISK_OID8)
+ {
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ ObjectId8GetDatum(toast_valueid));
+ }
else
- Assert(false);
+ {
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) toast_valueid));
+ }
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1921,8 +1967,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-09 21:53 Hannu Krosing <hannuk@google.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 72+ messages in thread
From: Hannu Krosing @ 2026-08-09 21:53 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi Michael
I understand from some earlier messages that in one of the patches you
refactored the code to make it easier to add new VARLENA on disk
types.
Where should I start looking for this?
-----
Hannu
On Thu, Aug 6, 2026 at 6:39 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Jul 30, 2026 at 05:01:02PM +0900, Yugo Nagata wrote:
> > On Thu, 30 Jul 2026 13:38:18 +0900 Yugo Nagata <nagata@sraoss.co.jp> wrote:
> > I also run a simple performance test. I measured the TPS on
> > the master branch and with the current patches applied under the
> > following conditions:
> >
> > Running pbench three times:
> >
> > $ pgbench test -f query.sql -c 8 -j 4 -T 100
> >
> > Results:
> >
> > master: 2686.229104, 2697.655096, 2687.894343
> > patched: 2685.292256, 2692.080214, 2690.005753
>
> Thanks for the test.
>
> > I haven't tested whether the previous version of the patches showed the
> > performance degradation that has been a concern, though.
>
> I am not sure that it is worth doing; I am not planning to get back to
> these versions, but it's true that it could be useful to see if your
> environment shows an impact of some impact. That's low priority, IMO.
>
> >> postgres=# CREATE TABLE tbl(t text) with (toast_value_type = 'x');
> >> ERROR: invalid value for enum option "toast_value_type": x
> >> DETAIL: Valid values are "oid".
> >>
> >> The detail message should be updated to include oid8.
>
> I had this one already fixed on my local branch.
>
> >> - 0009
> >>
> >> (1)
> >> The commit message says:
> >>
> >> TOAST pointers still rely on varatt_external and a single vartag, with
> >> all the values inserted in the bigint TOAST tables fed from the existing
> >> OID value generator.
> >>
> >> This made me a bit confused because chunk_ids insereted into TOAST tables using oid8
> >> are generated by GetNewObjectId8().
> >>
> >> + else if (toast_typid == OID8OID)
> >> + toast_pointer.va_valueid = GetNewObjectId8();
> >>
> >> This value is later cast to Oid here, so what is actually stored is a 4-byte chunk_id,
> >> but saying that the values are "fed from the existing OID value generator" seems
> >> a bit inaccurate to me.
>
> Okay, edited that a bit. I tend to rework all my commit messages
> before pushing, still you are right that this sounded a bit confusing.
>
> >> (2)
> >> /*
> >> - * Choose an OID to use as the value ID for this toast value.
> >> + * Choose a new value to use as the value ID for this toast value, be it
> >> + * for OID or int8-based TOAST relations.
> >>
> >> This patch relies on oid8, introduced by b139bd3b6ef, rather than int8, so, I
> >> wonder if this comment sould also be updated.
>
> Fixed.
>
> >> (3)
> >> belonging to the owning table. Every
> >> -<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
> >> -identifying the particular <acronym>TOAST</acronym>ed value),
> >> +<acronym>TOAST</acronym> table has the columns
> >> +<structfield>chunk_id</structfield> (an OID or an 8-byte integer identifying
> >> +the particular <acronym>TOAST</acronym>ed value),
> >> <structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
> >>
> >> Similarly, I think the documentation should mention oid8 rather than 8-byte
> >> integer. That would also be consistent with the CREATE TABLE documentation for
> >> the toast_value_type parameter.
>
> Hole in the docs, indeed.
>
> >> - 0011
> >>
> >> (4)
> >> + /*
> >> + * Check if this value already exists in the new toast
> >> + * table (corner case during table rewrite with multiple
> >> + * versions of the same row).
> >> + */
> >> + if (toastrel_valueid_exists(toastrel, va_valueid))
> >> + {
> >> + /* Match, so short-circuit the data storage loop below */
> >> + data_todo = 0;
> >> + }
> >>
> >> The same code appears later, together with a more detailed comment. How about
> >> moving the detailed comment here instead, or simply referring to
> >> the later comment?
>
> FWIW, I was looking again at this block of code, and reminded myself
> why this piece is this way: I want GetNewObjectId8(), InvalidOid8 and
> varatt_external_oid8 to have their own references in each block
> depending on the type of chunk_id. Perhaps a bit repetitive,
> efficient at least.
>
> >> (5)
> >> +/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
> >> +static inline bool
> >> +VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
> >> +{
> >> + return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
> >> }
> >>
> >> This macro is defined but not used. Is it intended to be used in places that
> >> check VARTAG_ONDISK_OID8 directly, for example:
> >>
> >> + if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
> >>
> >> ?
>
> Consistency with the other one, for extensions, as all the code paths
> checking for VARTAG_ONDISK_OID8 already make sure that we are dealing
> with an external pointer, so I did not see a point in checking
> VARATT_IS_EXTERNAL_ONDISK_OID8, that would make the extra
> VARATT_IS_EXTERNAL() check pointless.
>
> >> (6)
> >>
> >> any. Allowing for the varlena header bytes,
> >> -the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
> >> -bytes regardless of the actual size of the represented value.
> >> +the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
> >> +bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
> >> +when using an 8-byte integer, regardless of the actual size of the represented
> >> +value.
> >> </para>
> >>
> >> Should this also refer to Oid8 rather than 8-byte integner, for consistency?
>
> Fixed.
>
> > (7)
> >
> > Non-ASCII hyphen is used in this comment. This may not be prohibitted,
> > but I don't think it's common in the PostgreSQL code base.
>
> Seems like my keyboard has slipped a bit here. Will fix after
> replying to your other comments.
> --
> Michael
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-10 00:08 Michael Paquier <michael@paquier.xyz>
parent: Hannu Krosing <hannuk@google.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-08-10 00:08 UTC (permalink / raw)
To: Hannu Krosing <hannuk@google.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Aug 09, 2026 at 11:53:23PM +0200, Hannu Krosing wrote:
> I understand from some earlier messages that in one of the patches you
> refactored the code to make it easier to add new VARLENA on disk
> types.
(Top-posting?)
> Where should I start looking for this?
Quoting the top message of the thread:
```
The main feedback of the previous thread is that the previous
implementation with its callbacks for each vartag was not liked much,
and their were concerns with pointer redirections and
performance. This patch set uses what I am calling the "brutal"
approach, relying on a vartag_external of a varlena or the atttype of
the TOAST relation to decide which external toast pointer we should
use. This uses no function pointers, and patches the code to deal
with Oid or Oid8 TOAST values where it matters. So, this time,
performance cannot really be an issue.
```
The refactoring worked around a set of callbacks that could be
assigned to specific vartag_external, and one of the feedback that I
got is that this required function pointers, which could impact
performance for bulk-read or bulk-write of TOAST data. This
refactoring is gone as of round two, switching to direct if/elif
block. Overall the result is not that bad, as long as one is aware of
all the code paths that need to be handled on HEAD.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-10 07:37 Hannu Krosing <hannuk@google.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Hannu Krosing @ 2026-08-10 07:37 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Aug 10, 2026 at 2:08 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Sun, Aug 09, 2026 at 11:53:23PM +0200, Hannu Krosing wrote:
> > I understand from some earlier messages that in one of the patches you
> > refactored the code to make it easier to add new VARLENA on disk
> > types.
>
> (Top-posting?)
Not this time, just thought this was the right thread to ask general
questions :)
And I have now written me an Chrome extension which warns me when I am
top-posting in GMail by mistake, so should happen less frequently now.
> > Where should I start looking for this?
>
> Quoting the top message of the thread:
> ```
> The main feedback of the previous thread is that the previous
> implementation with its callbacks for each vartag was not liked much,
> and their were concerns with pointer redirections and
> performance. This patch set uses what I am calling the "brutal"
> approach, relying on a vartag_external of a varlena or the atttype of
> the TOAST relation to decide which external toast pointer we should
> use. This uses no function pointers, and patches the code to deal
> with Oid or Oid8 TOAST values where it matters. So, this time,
> performance cannot really be an issue.
> ```
Thanks, that explains why I could not find it.
I was hoping to us see move to using virtual function tables like in
all other places where we provide extensibility.
> The refactoring worked around a set of callbacks that could be
> assigned to specific vartag_external, and one of the feedback that I
> got is that this required function pointers, which could impact
> performance for bulk-read or bulk-write of TOAST data.
Are function pointers really that much of a concern considering that
we use tables of virtual pointers in core code paths like Table and
Index Access Methods?
Was this performance impact actually measured ?
> This refactoring is gone as of round two, switching to direct if/elif
> block.
Do I understand correctly that with this patch the selection between
oid and oid8 happens at the table level not at the individual toasted
item level ?
Maybe it would make sense to get better backwards compatibility and
smoother upgrades by doing the same I did in my direct toast patch to
allow both versions to live in the same table, namely to add a
separate oid8 column and use conditional indexes for both new oid8 and
old plain oid lookups?
This allows selecting the toast type at the time of creating the
toasted value and also to move back and forth between different toast
types at any time.
In direct toast the support for this kind of backwards compatibility required
* adding the colum to store the tid array replacing the index for
multi-block toast
* making the original toast index conditional on WHERE chunk_id IS NOT
NULL (or WHERE chunk_id <> 0) so that it would not need to generate
unneccessary oids
In case of oid8 it would be
* add a column for new chunk_id8 oid8
* add a conditional index on the (chunk_id8, chunk_seq) WHERE
chunk_id_oid8 IS NOT NULL
* making the original toast index conditional on WHERE chunk_id IS NOT NULL
-----------
Hannu
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-10 08:03 Michael Paquier <michael@paquier.xyz>
parent: Hannu Krosing <hannuk@google.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-08-10 08:03 UTC (permalink / raw)
To: Hannu Krosing <hannuk@google.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Aug 10, 2026 at 09:37:43AM +0200, Hannu Krosing wrote:
> Are function pointers really that much of a concern considering that
> we use tables of virtual pointers in core code paths like Table and
> Index Access Methods?
I am pretty sure that TOAST paths could get hotter than AMs under some
COPY-related workloads.
> Was this performance impact actually measured ?
I didn't measure any. Another comment I had was "overengineered",
which is perhaps fair as the TOAST code is quite linear in terms of
internals.
> Do I understand correctly that with this patch the selection between
> oid and oid8 happens at the table level not at the individual toasted
> item level ?
Yes, it's a design choice to use a table-level comparison. Choose one
with the reloption, then stick to it. Across upgrades, we keep the
same TOAST table. Rewrites are the same: no changes after a VACUUM
FULL, same atttype. There is nothing difference than what we do now,
which is why it's appealing to me.
> Maybe it would make sense to get better backwards compatibility and
> smoother upgrades by doing the same I did in my direct toast patch to
> allow both versions to live in the same table, namely to add a
> separate oid8 column and use conditional indexes for both new oid8 and
> old plain oid lookups?
Backward-compatibility and upgrades are handled in the patch set.
And TBH, I don't like much mixing multiple vartags in a single TOAST
table because it makes the whole layer more annoying to deal with.
Sticking with one atttype for each toast table is simpler. One
optimization would be to allow a OID external pointer to point to a
oid8 TOAST table, which offers benefits as long as we have less than 4
billion OIDs generated. This would not last long for some
deployments. Another idea that I found overkill was the use of a
per-toast sequence to generate the numbers. This can also be built on
top of what I have here, just found that expensive and a single oid8
counter in the control file is fine for all tables in a cluster
anyway.
> In case of oid8 it would be
> * add a column for new chunk_id8 oid8
> * add a conditional index on the (chunk_id8, chunk_seq) WHERE
> chunk_id_oid8 IS NOT NULL
> * making the original toast index conditional on WHERE chunk_id IS NOT NULL
Err. You're making TOAST tables larger than they should with
attributes that may not be required because some attributes may not be
required. With the TOAST data, it does not matter overall, but I'm
not much a fan of adding more data to disk than necessary, with more
indexes to maintain on each insert.
As a whole I'm not planning to go beyond what's presented on this
thread. Sticking to the simpler plan of having 8-byte values would
make a lot of customers I know of quite happy, because we have no exit
route now when it comes to core. (Spoiler: if I commit a patch, I own
its maintenance, and I'm well.. OK with what I have sent with a
long-term maintenance picture in mind.)
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-10 10:43 Hannu Krosing <hannuk@google.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Hannu Krosing @ 2026-08-10 10:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Aug 10, 2026 at 10:03 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Mon, Aug 10, 2026 at 09:37:43AM +0200, Hannu Krosing wrote:
> > Are function pointers really that much of a concern considering that
> > we use tables of virtual pointers in core code paths like Table and
> > Index Access Methods?
>
> I am pretty sure that TOAST paths could get hotter than AMs under some
> COPY-related workloads.
>
> > Was this performance impact actually measured ?
>
> I didn't measure any. Another comment I had was "overengineered",
> which is perhaps fair as the TOAST code is quite linear in terms of
> internals.
>
> > Do I understand correctly that with this patch the selection between
> > oid and oid8 happens at the table level not at the individual toasted
> > item level ?
>
> Yes, it's a design choice to use a table-level comparison. Choose one
> with the reloption, then stick to it. Across upgrades, we keep the
> same TOAST table. Rewrites are the same: no changes after a VACUUM
> FULL, same atttype. There is nothing difference than what we do now,
> which is why it's appealing to me.
>
> > Maybe it would make sense to get better backwards compatibility and
> > smoother upgrades by doing the same I did in my direct toast patch to
> > allow both versions to live in the same table, namely to add a
> > separate oid8 column and use conditional indexes for both new oid8 and
> > old plain oid lookups?
>
> Backward-compatibility and upgrades are handled in the patch set.
> And TBH, I don't like much mixing multiple vartags in a single TOAST
> table because it makes the whole layer more annoying to deal with.
Can you elaborate how having multiple vartags in the single table is
more annoying than having them in separate tables.
My experience is that on insert you select the vartag to use from a
GUC or table attributem and on read we already have a number of
different vartags each requiring special handling.
> Sticking with one atttype for each toast table is simpler. One
> optimization would be to allow a OID external pointer to point to a
> oid8 TOAST table, which offers benefits as long as we have less than 4
> billion OIDs generated. This would not last long for some
> deployments.
Agreed, but we also never need more than 5 bytes worth of space with
current 8 kb pages, as there would be not toast table rows to point to
:)
As you are concerned about space usage, but want to keep accessing
toast chunks via indexes, have you considered something like the
43-bit integers used to represent tuple ids in GIN posting lists?
> Another idea that I found overkill was the use of a
> per-toast sequence to generate the numbers. This can also be built on
> top of what I have here, just found that expensive and a single oid8
> counter in the control file is fine for all tables in a cluster
> anyway.
I tend to agree. Theoretically it could result in better locking
behaviour between the sequences, but in practice it would be very
unlikely to have a large number of toast tables in active parallel
use.
And the overhead of index inserts will always be larger than getting
the number from sequence, so one sequence should be good for at least
10 tables. And it is unlikely to need more.
> > In case of oid8 it would be
> > * add a column for new chunk_id8 oid8
> > * add a conditional index on the (chunk_id8, chunk_seq) WHERE
> > chunk_id_oid8 IS NOT NULL
> > * making the original toast index conditional on WHERE chunk_id IS NOT NULL
>
> Err. You're making TOAST tables larger than they should with
> attributes that may not be required because some attributes may not be
> required.
The overhead is only in the the pg_attribute table, as we already have
the 8 bits of null bitmnap in the header and this will not grow, and
any values not used will not be stored.
And in case of direct toast storing (potentially compressed) tid lists
will be way more space efficient than having an index, in addition to
having much better performance.
In direct toas case it can even result in (very slighly) smaller space
usage because of not storing oid, and at least in the case of single
chunk toast also not he chunk_seq.
> With the TOAST data, it does not matter overall, but I'm
> not much a fan of adding more data to disk than necessary, with more
> indexes to maintain on each insert.
The benefit of my scheme is that you do not need to maintain the
indexes you are not using.
> As a whole I'm not planning to go beyond what's presented on this
> thread. Sticking to the simpler plan of having 8-byte values would
> make a lot of customers I know of quite happy, because we have no exit
> route now when it comes to core.
I still hope to convince you that an even simpler plan to get all the
benefits of larger id space by using tids directly :)
The patch is much simpler and mostly just bypasses code paths that are used.
At high level it
For select
- goes directly to toast tuple, bypassing index lookup with all its
locking and pinning
- if there are more than a single chunk it hets the list of tuple ids,
again bypassing the index lookups, and loads them directly
- if there are larger number of chunks than can fit their tids in one
array, theis is repeated recursively for some of the toast tuples
fetched using this list
At insert/update time
- it bypasses getting an OID
- it inserts the chunks exactly as the current code
- it bypasses index operations, opening, pinning, adding the entry,
poissibly splitting index page, etc.
At vacuum time (not implemented yet) it can bypass collecting the
tuple id for index cleanup for deleted tuples
So the only thing _added_ is collecting the chunk ids in an array and
storing it in selected end-of-run pages instead of insering them in
the toast index.
Everything else streamlines the code bypassing expensive operations.
(And I know adding tid array messes with fitting exacly 4 chunks of
~2000 bytes on each page, but longer term plan is to relax this by
also adding another array of chunk offsets togetehr with the tid array
which would allow using all the page space in toast pages. Also the
tid array is highly compressible if the tuples are allocated together,
as shown in GIN posting lists. And I have a separate patch that would
improve the compression another 6x over GIN posting lists for common
cases, totaling 20-30 x compression)
The main thing (that I know of) that I have not yet done handling
VACUUM FULL and CLUSTER for direct toast tables.
Disabling them if direct toast tuples exist is trivial.
Supporting explicit CLUSTER / VACUUM FULL on main table which also
handles toast would need some work.
Just clustering the toast table only in OID order can be useless for
performance if the toasted columns have been frequently updated or if
data has been added later to some.
Currently this can be done manually using CREATE TABLE (LIKE oldtable
INCLUDING ALL) + INSERT INTO ... SELECT and then swapping the table
but it would be much nicer to have it integrated as CLUSTER / VACUUM
FULL
This would also be what the new
Because this is what the new REPACK command does anyway I am not sure
that also supporting CLUSTER AND VACUUM to include TOAST reqrite is
worth doing.
I am still investigating your patch set to understand if I missed any
other parts that would need changes.
---
Hannu
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-08-13 06:38 Michael Paquier <michael@paquier.xyz>
parent: Hannu Krosing <hannuk@google.com>
0 siblings, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-08-13 06:38 UTC (permalink / raw)
To: Hannu Krosing <hannuk@google.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Aug 10, 2026 at 12:43:37PM +0200, Hannu Krosing wrote:
> The main thing (that I know of) that I have not yet done handling
> VACUUM FULL and CLUSTER for direct toast tables.
>
> Disabling them if direct toast tuples exist is trivial.
Well, you say that. Noted. Now, it also sounds to me that you are
talking about oranges and I am talking about apples, which are two
different things. There is nothing that prevents doing one thing and
the other (I write that after looking at your AI-generated code that
you have mentioned as not reading yourself, yes). Here the apples are
about one full patch set that's posted on this thread, split for
easier review, able to handle all cases we care about in core
including backward-compatibility, relation rewrites, upgrades,
dump/restore, versus oranges clearly presented as partially complete
without a clear complete picture.
Honestly, and to keep it short, I don't see enough arguments to be
convinced that the oranges are better than the apples. Having the
code presenting the oranges to be fully generated by an LLM agent
does not help IMO, either.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-02 20:27 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-02 20:27 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Wed, Aug 5, 2026 at 9:39 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> Seems like my keyboard has slipped a bit here. Will fix after
> replying to your other comments.
Thanks Michael for working on this feature.
I started looking at the patches and TOAST code. Here's my take.
I understand that when 4-byte chunk_id exhaustion happens, query
(insert/update) latencies can increase from milliseconds to minutes
and hours, or even days as OID use approaches the 4 billion limit.
Although this seems to happen rarely (insert- or update-heavy
workloads with more toastable columns, i.e., more chunk_id use per
row, or dead tuples in the TOAST table when vacuum is busy or yet to
get to the TOAST table and index), when it does happen, the mitigation
for the customer seems hard: partition the table or fix vacuum.
I believe that 8-byte chunk_id is the right direction to solve this
problem in a simpler way, without affecting existing 4-byte chunk_id
tables across upgrades and providing a simple path for customers to
migrate to 8-byte chunk_id. My reading of the patches says that it
follows these principles.
I first tried to reproduce the query latency increase problem with
less time and disk space. Here are the results. I ensured each insert
steps over K ids using pg_resetwal, which equals the number of
iterations spent in GetNewOidWithIndex(). There's a clear benefit as
the 8-byte chunk_id avoids the retries while getting the new OID, and
since a 64-bit OID is almost never exhausted, this works. Attached the
test script that I used for reference.
K HEAD_INSERT patched_INSERT
1,000,000 1.25 s 1.0 ms
10,000,000 12.4 s 1.0 ms
50,000,000 1 min 1.0 ms
100,000,000 2 min 1.0 ms
200,000,000 9 min 0.9 ms
With this context, I started to review the patches. One thing I liked
is the way the patches were split; they made review a lot easier. Here
are some comments.
Comments on 0001:
1/
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
While changing Oid to Oid8 is fine for internal functions, is it okay
to change it for the table_relation_fetch_toast_slice table AM? Is
there any chance that an external table AM forgets to update? I'm not
arguing against this change, just checking if there's any way out
here. Of course, the compiler will generate an incompatible function
parameter type warning, and since the change is going in a major
release, that seems fine. One idea could be to have a TOAST version
field elsewhere to detect such changes, but that seems overkill. So,
having it like 0001 seems fine to me unless others have any thoughts.
PS: one option is to send the 8-byte OID via a caller-allocated
varlena result pointer, but this seems a bit ugly.
2/
@@ -4997,7 +4997,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb,
ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
Widening the per-toast-chunk hash key to an 8-byte OID still handles
4-byte chunk_ids correctly during logical decoding, since the smaller
values zero-extend. The tradeoff is that with an 8-byte key the hash
no longer uses the uint32_hash fast path and falls back to tag_hash,
so even existing 4-byte chunk_id tables lose that optimization during
logical decoding. I haven't measured the effect, but it seems worth
checking.
3/ Rest of the 0001 changes look good to me, although they use an
8-byte format specifier for error reporting, which can still work for
4-byte chunk_ids.
I think 0002 and 0003 are purely mechanical; they look good to me and
can go in first.
Comments on 0004:
1/
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value); + WHERE
o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
Changing the return value of pg_column_toast_chunk_id() to 8-byte OID
is fine and it can work for existing 4-byte chunk_ids. Do we need the
above typecasting in tests given the chunk_id is captured from
pg_column_toast_chunk_id() while creating the table?
Comments on 0005:
1/ + return murmurhash64(DatumGetObjectId8(datum));
I think having catalog cache support for 8-byte OID might be useful on
its own and it looks good to me except the following: can we typecast
the Oid8 to murmurhash64((uint64) DatumGetObjectId8(datum))?
Comments on 0008:
1/ I think adding reloption seems okay to me, so the 0008 patch looks
good. Just one suggestion: having a GUC to set it once for all new
tables seems a good idea as it avoids application changes, but
starting with reloption is good enough.
I will take a look at 0006, 0007, 0009, 0010, 0011 in the coming weeks.
One more thought: existing tables won't get 8-byte chunk_ids as part
of pg_upgrade. pg_dump and pg_restore could be used after the upgrade
for existing customers to get their tables onto this format. At some
point, providing concurrent repack-like support (or reusing the
underlying machinery) to do this online would be nice to have.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-03 04:58 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 2 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-03 04:58 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Sep 02, 2026 at 01:27:00PM -0700, Bharath Rupireddy wrote:
> I first tried to reproduce the query latency increase problem with
> less time and disk space. Here are the results. I ensured each insert
> steps over K ids using pg_resetwal, which equals the number of
> iterations spent in GetNewOidWithIndex(). There's a clear benefit as
> the 8-byte chunk_id avoids the retries while getting the new OID, and
> since a 64-bit OID is almost never exhausted, this works. Attached the
> test script that I used for reference.
>
> K HEAD_INSERT patched_INSERT
> 1,000,000 1.25 s 1.0 ms
> 10,000,000 12.4 s 1.0 ms
> 50,000,000 1 min 1.0 ms
> 100,000,000 2 min 1.0 ms
> 200,000,000 9 min 0.9 ms
This measures how much time we take to grab a free OID value, hoping
that there is a hole.
> With this context, I started to review the patches. One thing I liked
> is the way the patches were split; they made review a lot easier. Here
> are some comments.
Thanks for the comments.
> Comments on 0001:
>
> 1/
> +table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
> While changing Oid to Oid8 is fine for internal functions, is it okay
> to change it for the table_relation_fetch_toast_slice table AM? Is
> there any chance that an external table AM forgets to update? I'm not
> arguing against this change, just checking if there's any way out
> here. Of course, the compiler will generate an incompatible function
> parameter type warning, and since the change is going in a major
> release, that seems fine. One idea could be to have a TOAST version
> field elsewhere to detect such changes, but that seems overkill. So,
> having it like 0001 seems fine to me unless others have any thoughts.
> PS: one option is to send the 8-byte OID via a caller-allocated
> varlena result pointer, but this seems a bit ugly.
Yes, assuming that a table AM uses its own fetch_toast_slice(), it
will need to update probably for its own ScanKeyInit() if an external
TOAST table is in use for the OID8 case, assuming that they need to do
so based on the reloption for the toast value type. I am not really
convinced that we need to be fancy here, telling that after looking at
out-of-code projets that include their own callbacks:
https://github.com/eatonphil/pgtam (no support)
https://github.com/rohankumardubey/pg_mooncake
https://github.com/neurdb/neurdb (no support)
https://github.com/duckdb/pg_duckdb (totally different callback here)
https://github.com/timescale/timescaledb
https://github.com/jeffreydwalter/pg_tde/ (needs refresh)
https://github.com/ewhauser/unitpg/ (just reuses the heap call)
There are also a few more, but I am seeing nothing beyond an extra
ScanKeyInit() update required, in a sea of non-supported cases.
> Widening the per-toast-chunk hash key to an 8-byte OID still handles
> 4-byte chunk_ids correctly during logical decoding, since the smaller
> values zero-extend. The tradeoff is that with an 8-byte key the hash
> no longer uses the uint32_hash fast path and falls back to tag_hash,
> so even existing 4-byte chunk_id tables lose that optimization during
> logical decoding. I haven't measured the effect, but it seems worth
> checking.
Yeah, point taken. I'm putting a note down on this one.
> I think 0002 and 0003 are purely mechanical; they look good to me and
> can go in first.
Thanks. 0002 and 0003 are kind of independent on the rest, not
requiring any Oid8 idea, so I'll go apply them.
> 1/
> - WHERE o.chunk_id != pg_column_toast_chunk_id(c.value); + WHERE
> o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
>
> Changing the return value of pg_column_toast_chunk_id() to 8-byte OID
> is fine and it can work for existing 4-byte chunk_ids. Do we need the
> above typecasting in tests given the chunk_id is captured from
> pg_column_toast_chunk_id() while creating the table?
I guess we do. Will double-check.
> Comments on 0005:
>
> 1/ + return murmurhash64(DatumGetObjectId8(datum));
>
> I think having catalog cache support for 8-byte OID might be useful on
> its own and it looks good to me except the following: can we typecast
> the Oid8 to murmurhash64((uint64) DatumGetObjectId8(datum))?
Ah, you want to add an extra cast here, with (uint64) for the
murmurhash64. Hmm why not, that seems more consistent with the rest
of the area. This one is also a rather independent item for the Oid8,
so I'm also planning to apply it to reduce the stack.
> Comments on 0008:
>
> 1/ I think adding reloption seems okay to me, so the 0008 patch looks
> good. Just one suggestion: having a GUC to set it once for all new
> tables seems a good idea as it avoids application changes, but
> starting with reloption is good enough.
The round one of the patch set used a GUC. Andres has argued in favor
of a reloption.
> I will take a look at 0006, 0007, 0009, 0010, 0011 in the coming weeks.
Cool, thanks.
> One more thought: existing tables won't get 8-byte chunk_ids as part
> of pg_upgrade. pg_dump and pg_restore could be used after the upgrade
> for existing customers to get their tables onto this format. At some
> point, providing concurrent repack-like support (or reusing the
> underlying machinery) to do this online would be nice to have.
The infrastructure that could be used to switch the toast value type
and its rewrite is out of scope, the patch being complicated enough..
We could always think about that later, giving the choice in core is
much more important to me as a first step, because without the
OID8/OID choice, there is no discussion about the rewrite part.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-04 04:28 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-04 04:28 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Sep 03, 2026 at 01:58:19PM +0900, Michael Paquier wrote:
> > I think 0002 and 0003 are purely mechanical; they look good to me and
> > can go in first.
>
> Thanks. 0002 and 0003 are kind of independent on the rest, not
> requiring any Oid8 idea, so I'll go apply them.
0002 has been applied as of 2b3d11aaed89. 0003 does a bunch of
renames and in order to raise a bit more awareness and not get
screamed at after-the-fact, I have posted that on a different thread:
https://www.postgresql.org/message-id/apoD1Ulet2PtDrRV@paquier.xyz
The first feedbacks I had we OK, so I don't expect any pushback, just
being careful and we still have time.
After an extra set of checks, I am dropping the catcache thing for
OID8OID, for the simple reason that there is no need for it anymore.
I am pretty sure that I did some OID8 catalog lookup at some point of
this patch set, but nothing shows up now. Perhaps that was in the
round 1 of the discussion.
I'll post a rebase of the remaining pieces once the rename thread is
completely settled, hopefully around the beginning of next week.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-05 03:11 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-05 03:11 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Wed, Sep 2, 2026 at 9:58 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Sep 02, 2026 at 01:27:00PM -0700, Bharath Rupireddy wrote:
> > I first tried to reproduce the query latency increase problem with
> > less time and disk space. Here are the results. I ensured each insert
> > steps over K ids using pg_resetwal, which equals the number of
> > iterations spent in GetNewOidWithIndex(). There's a clear benefit as
> > the 8-byte chunk_id avoids the retries while getting the new OID, and
> > since a 64-bit OID is almost never exhausted, this works. Attached the
> > test script that I used for reference.
> >
> > K HEAD_INSERT patched_INSERT
> > 1,000,000 1.25 s 1.0 ms
> > 10,000,000 12.4 s 1.0 ms
> > 50,000,000 1 min 1.0 ms
> > 100,000,000 2 min 1.0 ms
> > 200,000,000 9 min 0.9 ms
>
> This measures how much time we take to grab a free OID value, hoping
> that there is a hole.
Yes, not all the chunk_ids are occupied in this testing, so there are
holes. I wanted to keep the testing time to a minimum and show how
long it usually takes for insert/update queries to fetch a chunk_id
when it's closer to the 4 billion limit. I attached the test script
used for this testing (courtesy: Claude Code). Do you have any other
test cases in mind?
> > Comments on 0001:
> >
> > 1/
> > +table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
> > While changing Oid to Oid8 is fine for internal functions, is it okay
> > to change it for the table_relation_fetch_toast_slice table AM? Is
> > there any chance that an external table AM forgets to update? I'm not
> > arguing against this change, just checking if there's any way out
> > here. Of course, the compiler will generate an incompatible function
> > parameter type warning, and since the change is going in a major
> > release, that seems fine. One idea could be to have a TOAST version
> > field elsewhere to detect such changes, but that seems overkill. So,
> > having it like 0001 seems fine to me unless others have any thoughts.
> > PS: one option is to send the 8-byte OID via a caller-allocated
> > varlena result pointer, but this seems a bit ugly.
>
> Yes, assuming that a table AM uses its own fetch_toast_slice(), it
> will need to update probably for its own ScanKeyInit() if an external
> TOAST table is in use for the OID8 case, assuming that they need to do
> so based on the reloption for the toast value type. I am not really
> convinced that we need to be fancy here, telling that after looking at
> out-of-code projets that include their own callbacks:
> https://github.com/eatonphil/pgtam (no support)
> https://github.com/rohankumardubey/pg_mooncake
> https://github.com/neurdb/neurdb (no support)
> https://github.com/duckdb/pg_duckdb (totally different callback here)
> https://github.com/timescale/timescaledb
> https://github.com/jeffreydwalter/pg_tde/ (needs refresh)
> https://github.com/ewhauser/unitpg/ (just reuses the heap call)
>
> There are also a few more, but I am seeing nothing beyond an extra
> ScanKeyInit() update required, in a sea of non-supported cases.
Thanks for providing these details. It may help table AM implementers
to have a note in the commit message, the docs, or this thread
explaining how they need to adapt.
> > Widening the per-toast-chunk hash key to an 8-byte OID still handles
> > 4-byte chunk_ids correctly during logical decoding, since the smaller
> > values zero-extend. The tradeoff is that with an 8-byte key the hash
> > no longer uses the uint32_hash fast path and falls back to tag_hash,
> > so even existing 4-byte chunk_id tables lose that optimization during
> > logical decoding. I haven't measured the effect, but it seems worth
> > checking.
>
> Yeah, point taken. I'm putting a note down on this one.
Also, I think there's an issue in ReorderBufferToastAppendChunk() with
0001. The chunk_id is still fetched as a 4-byte value, so past 4
billion chunk_ids the truncated key stored in the TOAST hash won't
match the 64-bit value used for the lookup later in
ReorderBufferToastReplace().
> > Comments on 0008:
> >
> > 1/ I think adding reloption seems okay to me, so the 0008 patch looks
> > good. Just one suggestion: having a GUC to set it once for all new
> > tables seems a good idea as it avoids application changes, but
> > starting with reloption is good enough.
>
> The round one of the patch set used a GUC. Andres has argued in favor
> of a reloption.
That works for me. A quick question, though I haven't tested this.
Does ALTER TABLE error out if the reloption is changed on an existing
table?
> > I will take a look at 0006, 0007, 0009, 0010, 0011 in the coming weeks.
>
> Cool, thanks.
>
> > One more thought: existing tables won't get 8-byte chunk_ids as part
> > of pg_upgrade. pg_dump and pg_restore could be used after the upgrade
> > for existing customers to get their tables onto this format. At some
> > point, providing concurrent repack-like support (or reusing the
> > underlying machinery) to do this online would be nice to have.
>
> The infrastructure that could be used to switch the toast value type
> and its rewrite is out of scope, the patch being complicated enough..
> We could always think about that later, giving the choice in core is
> much more important to me as a first step, because without the
> OID8/OID choice, there is no discussion about the rewrite part.
That's fine by me. That said, it would be good to document at least
one clear way for existing users to move their tables to 8-byte TOAST
chunk_ids, presumably pg_dump and pg_restore after setting the
reloption.
I'm also wondering if pg_dump needs an option to generate table
schemas with the new reloption set, so that users don't have to edit
the dump file. Editing may not always be possible (for example, with
the custom and directory formats). This can be a follow-up patch, but
having at least one supported way to migrate existing tables seems
important to me.
On Thu, Sep 3, 2026 at 9:28 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> After an extra set of checks, I am dropping the catcache thing for
> OID8OID, for the simple reason that there is no need for it anymore.
> I am pretty sure that I did some OID8 catalog lookup at some point of
> this patch set, but nothing shows up now. Perhaps that was in the
> round 1 of the discussion.
Agreed, dropping it seems right at a quick glance. The chunk_id comes
from the heap row's TOAST pointer and is used as a scan key against
the TOAST index, so nothing looks up an OID8 value through the catalog
cache.
> I'll post a rebase of the remaining pieces once the rename thread is
> completely settled, hopefully around the beginning of next week.
Thanks.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
Attachments:
nocfbot_toast_oid.sh (application/x-sh, 5.5K) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-07 07:54 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-07 07:54 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Sep 04, 2026 at 08:11:00PM -0700, Bharath Rupireddy wrote:
> On Wed, Sep 2, 2026 at 9:58 PM Michael Paquier <michael@paquier.xyz> wrote:
>> On Wed, Sep 02, 2026 at 01:27:00PM -0700, Bharath Rupireddy wrote:
>>> Widening the per-toast-chunk hash key to an 8-byte OID still handles
>>> 4-byte chunk_ids correctly during logical decoding, since the smaller
>>> values zero-extend. The tradeoff is that with an 8-byte key the hash
>>> no longer uses the uint32_hash fast path and falls back to tag_hash,
>>> so even existing 4-byte chunk_id tables lose that optimization during
>>> logical decoding. I haven't measured the effect, but it seems worth
>>> checking.
While navigating through the code, I am not planning to worry about
it at the end, as the performance sits close to some WAL record
handling and tuple copying, where most of the performance is on the
table there.
> Also, I think there's an issue in ReorderBufferToastAppendChunk() with
> 0001. The chunk_id is still fetched as a 4-byte value, so past 4
> billion chunk_ids the truncated key stored in the TOAST hash won't
> match the 64-bit value used for the lookup later in
> ReorderBufferToastReplace().
Hmm. This one may be debatable because it would not be reached until
the OID8 business is plugged into the TOAST values. Adding that in
0001 is less business to deal with after. For now I've fixed that in
0001 to keep the changes with reorderbuffer.c more isolated.
> That works for me. A quick question, though I haven't tested this.
> Does ALTER TABLE error out if the reloption is changed on an existing
> table?
I was not doing that with only simplicity in mind, so one could change
toast_value_type after the initial TOAST table creation, with one
argument in mind but I agree that it may sounds a bit weird: dumps.
One could change the reloption, do a dump, and re-create the relation
with a different TOAST layer. Agreed that it's a bit twisted but we
don't have a lot of precedent for this kind of choices with
reloptions. I recall one similar case in the backend code, with BRIN
indexes and pages_per_range (?) that do not affect post-creation
patterns. The WITH added by the dump is kind of nice for re-creation
based on raw data.
There is also a second scenario where this can be useful:
CREATE TABLE t (a int);
ALTER TABLE t SET (toast_value_type = oid8);
ALTER TABLE t ADD COLUMN b text; -- creates TOAST relation
I've mentioned that in the docs already, as of the "creating":
+ The toast_value_type specifies the attribute type of
+ <literal>chunk_id</literal> used when initially creating a toast
+ relation for this table
I have added some tests for ALTER TABLE with some rewrites, though.
That felt like a hole.
> That's fine by me. That said, it would be good to document at least
> one clear way for existing users to move their tables to 8-byte TOAST
> chunk_ids, presumably pg_dump and pg_restore after setting the
> reloption.
Hmm, yeah. Perhaps. The dump/restore trick is one, at least. Not
sure where adding this mention would be best, and if we actually need
to add that. :)
> I'm also wondering if pg_dump needs an option to generate table
> schemas with the new reloption set, so that users don't have to edit
> the dump file. Editing may not always be possible (for example, with
> the custom and directory formats). This can be a follow-up patch, but
> having at least one supported way to migrate existing tables seems
> important to me.
The best thing that I came up for that was a GUC, because you'd want
a dump/restore option able to say "I don't want the TOAST type" to be
dumped or restored, able to enforce a type across the board for a
database, a namespace or more than N objects at the same time. The
GUC idea was shut down in round one by Andres.
> Agreed, dropping it seems right at a quick glance. The chunk_id comes
> from the heap row's TOAST pointer and is used as a scan key against
> the TOAST index, so nothing looks up an OID8 value through the catalog
> cache.
Following c5806f2165b5 and 2b3d11aaed89, attached is a rebase of the
rest.
While on it, I have added one StaticAssertDecl() to enforce the
no-padding rule of vartag_external_oid8. There was also a spot I have
missed in pg_column_toast_chunk_id() to handle oid8 toast pointers, in
the last patch.
On top of that, while doing more review, there was also an issue that
I got on my notes for some time but completely forgot about in this
thread: toast_tuple_find_biggest_attribute() needs some relcaching for
the toast value type to enforce a correct decision, or we may be off
in deciding if something should be compressed or not depending on the
toast table we are dealing with. This uses the OID's max chunk size
as a safety measure, which should be OK. This part is added on top of
the rest, as 0009.
Finally, I have asked Claude for a round of reviews for the full patch
set, and it has spotted two stupid mistakes caused by the new vartag
for oid8, where I missed its handling:
- toast_get_compression_id() missed handling for oid8 external
pointers.
- toast_tuple_init() can compare two external TOAST pointers, but we
cannot memcmp() safely when dealing with tuples with different
vartags, the oid8 one being larger than the oid one. That's a nice
catch, for an egde case when doing cross-updates and TOAST pointer
reuse.
Attached is v14, with all that addressed (should perhaps have moved
the tests in 0008 to 0007, but it's late here and that does not change
the stuff to review).
--
Michael
Attachments: download all (10 files, .tar.gz)
v14-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch (text/plain, 16.1K) download
From 9505b2b8b90064ff8de86024d399655ac2ed1c71 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 13:40:41 +0900
Subject: [PATCH v14 1/9] Refactor some TOAST value ID code to use Oid8 instead
of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 23 +++++---
contrib/amcheck/verify_heapam.c | 55 +++++++++++--------
6 files changed, 60 insertions(+), 44 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index b2346fd75bd4..aaae6e0fef69 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index ff03a2b816fc..ebec31a22e40 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1987,7 +1987,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 0f234661bfa9..ca5ed8dfabca 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);
-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)
+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)
+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 3b7c658a90e6..81154c17376c 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -729,7 +729,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",
+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -741,13 +741,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -756,7 +756,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",
+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -785,7 +785,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",
+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 457d544eec93..dd4b6e8f7f45 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */
+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -5010,7 +5010,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5034,8 +5034,10 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;
+ Oid8 chunk_id;
int32 chunk_seq;
+ Oid valueid_type;
+ Datum valueid_datum;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
@@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));
+ valueid_type = TupleDescAttr(desc, 0)->atttypid;
+ valueid_datum = fastgetattr(newtup, 1, desc, &isnull);
+ if (valueid_type == OID8OID)
+ chunk_id = DatumGetObjectId8(valueid_datum);
+ else
+ chunk_id = DatumGetObjectId(valueid_datum);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
@@ -5061,11 +5068,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5174,6 +5181,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5194,13 +5202,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,
+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index bbaaec89f8ae..33b00fa4dc5c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1564,9 +1564,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 chunksize;
int32 expected_size;
int32 max_chunk_size;
+ Oid8 toast_valueid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1574,16 +1576,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",
+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",
+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1594,8 +1596,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",
+ toast_valueid,
chunk_seq));
return;
}
@@ -1614,8 +1616,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",
+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1626,8 +1628,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",
+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1637,8 +1639,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",
+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1669,6 +1671,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external_oid toast_pointer;
@@ -1777,12 +1780,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",
- toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1809,16 +1813,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",
- toast_pointer.va_valueid, cmid));
+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",
+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",
+ toast_pointer_valueid));
return true;
}
@@ -1826,8 +1830,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",
+ toast_pointer_valueid));
return true;
}
@@ -1873,6 +1877,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1903,14 +1908,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;
+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " not found in toast table",
+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",
+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.55.0
v14-0002-Switch-pg_column_toast_chunk_id-return-value-fro.patch (text/plain, 5.7K) download
From 9ecff1e6548da6d086d6d562c702140129312f6e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v14 2/9] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6979c7d11613..2b65308eaff5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7910,7 +7910,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',
+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index e5632d930682..1e264904a6e1 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);
+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out
index cb14ddcee34b..21859fc0dce2 100644
--- a/src/test/isolation/expected/cluster-toast-value-reuse.out
+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out
@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec
index 9a2d10600b39..96cf899db296 100644
--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec
+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec
@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
index fa76bcf28ef9..cc8f034d0167 100644
--- a/src/test/modules/injection_points/specs/repack_toast.spec
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);
+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2990e0c4f286..35d6ee2ce9eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 950d9ab1a4a1..94b91964c046 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 54eeb42e5bce..70d762e73328 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>
+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.55.0
v14-0003-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch (text/plain, 9.3K) download
From a87a368c80749a840d68f13f19c984f927ffc4a4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 10:57:59 +0900
Subject: [PATCH v14 3/9] Add support for TOAST chunk_id type in binary
upgrades
This commit adds a new function, which would set the type of a chunk_id
attribute for a TOAST table across upgrades. This piece currently works
only with chunk_id = OIDOID, but it is required in a follow-up patch
where support for chunk_id = OID8OID is supported on top of the existing
one.
---
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 ++++
src/backend/catalog/heap.c | 1 +
src/backend/catalog/toasting.c | 20 ++++++++++++++++++-
src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++++
src/bin/pg_dump/pg_dump.c | 10 +++++++++-
.../expected/spgist_name_ops.out | 6 ++++--
7 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae443859..117f3e3f7746 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -29,6 +29,7 @@ extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_index_pg_class_relfilenumber;
extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber;
+extern PGDLLIMPORT Oid binary_upgrade_next_toast_chunk_id_typoid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2b65308eaff5..21357fccf8ba 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11995,6 +11995,10 @@
proname => 'binary_upgrade_set_next_toast_pg_class_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_toast_pg_class_oid' },
+{ oid => '8219', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_toast_chunk_id_typoid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_toast_chunk_id_typoid' },
{ oid => '3589', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_set_next_pg_enum_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index b018f26545b6..9094a2ab77fb 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -80,6 +80,7 @@
/* Potentially set by pg_upgrade_support functions */
Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid;
Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid;
+Oid binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
RelFileNumber binary_upgrade_next_heap_pg_class_relfilenumber = InvalidRelFileNumber;
RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber = InvalidRelFileNumber;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 4aa52a4bd253..bc0a72268335 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -146,6 +146,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_chunkid_typid = OIDOID;
/*
* Is it already toasted?
@@ -184,6 +185,23 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!OidIsValid(binary_upgrade_next_toast_pg_class_oid))
return false;
+
+ /*
+ * The attribute type for chunk_id should have been set when requesting
+ * a TOAST table creation.
+ */
+ if (!OidIsValid(binary_upgrade_next_toast_chunk_id_typoid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("toast chunk_id type not set while in binary upgrade mode")));
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
+ binary_upgrade_next_toast_chunk_id_typoid)));
+
+ toast_chunkid_typid = binary_upgrade_next_toast_chunk_id_typoid;
+ binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
}
/*
@@ -205,7 +223,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,
+ toast_chunkid_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4feeb..06b789d71ded 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -149,6 +149,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_toast_chunk_id_typoid(PG_FUNCTION_ARGS)
+{
+ Oid typoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_toast_chunk_id_typoid = typoid;
+
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b5b257f9983c..9a8a824d1bfb 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -104,6 +104,7 @@ typedef struct
RelFileNumber relfilenumber; /* object filenode */
Oid toast_oid; /* toast table OID */
RelFileNumber toast_relfilenumber; /* toast table filenode */
+ Oid toast_chunk_id_typoid; /* type of chunk_id attribute */
Oid toast_index_oid; /* toast table index OID */
RelFileNumber toast_index_relfilenumber; /* toast table index filenode */
} BinaryUpgradeClassOidItem;
@@ -5872,7 +5873,10 @@ collectBinaryUpgradeClassOids(Archive *fout)
const char *query;
query = "SELECT c.oid, c.relkind, c.relfilenode, c.reltoastrelid, "
- "ct.relfilenode, i.indexrelid, cti.relfilenode "
+ "ct.relfilenode, i.indexrelid, cti.relfilenode, "
+ "(SELECT a.atttypid FROM pg_attribute AS a "
+ " WHERE a.attrelid = c.reltoastrelid AND attname = 'chunk_id'::text) "
+ " AS toastchunktypid "
"FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_index i "
"ON (c.reltoastrelid = i.indrelid AND i.indisvalid) "
"LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) "
@@ -5894,6 +5898,7 @@ collectBinaryUpgradeClassOids(Archive *fout)
binaryUpgradeClassOids[i].toast_relfilenumber = atooid(PQgetvalue(res, i, 4));
binaryUpgradeClassOids[i].toast_index_oid = atooid(PQgetvalue(res, i, 5));
binaryUpgradeClassOids[i].toast_index_relfilenumber = atooid(PQgetvalue(res, i, 6));
+ binaryUpgradeClassOids[i].toast_chunk_id_typoid = atooid(PQgetvalue(res, i, 7));
}
PQclear(res);
@@ -5958,6 +5963,9 @@ binary_upgrade_set_pg_class_oids(Archive *fout,
appendPQExpBuffer(upgrade_buffer,
"SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n",
entry->toast_relfilenumber);
+ appendPQExpBuffer(upgrade_buffer,
+ "SELECT pg_catalog.binary_upgrade_set_next_toast_chunk_id_typoid('%u'::pg_catalog.oid);\n",
+ entry->toast_chunk_id_typoid);
/* every toast table has an index */
appendPQExpBuffer(upgrade_buffer,
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede2430..35e59d0cd83c 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -61,9 +61,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -110,9 +111,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.55.0
v14-0004-Enlarge-OID-generation-to-8-bytes.patch (text/plain, 14.5K) download
From f61dbd2f4dd090b5864911eb8afdd6302dcfca18 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:15:15 +0900
Subject: [PATCH v14 4/9] Enlarge OID generation to 8 bytes
This adds a new routine called GetNewObjectId8() in varsup.c, which is
able to retrieve a 64b OID. GetNewObjectId() is kept compatible with
its origin, where we still check that the lower 32 bits of the counter
do not wraparound, handling the FirstNormalObjectId case.
pg_resetwal -o/--next-oid is updated to be able to handle 8-byte OIDs.
---
src/include/access/transam.h | 3 +-
src/include/access/xlog.h | 2 +-
src/include/catalog/pg_control.h | 2 +-
src/backend/access/rmgrdesc/xlogdesc.c | 8 +--
src/backend/access/transam/varsup.c | 62 ++++++++++++++++-------
src/backend/access/transam/xlog.c | 8 +--
src/backend/access/transam/xlogrecovery.c | 2 +-
src/bin/pg_controldata/pg_controldata.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 10 ++--
doc/src/sgml/ref/pg_resetwal.sgml | 6 +--
10 files changed, 66 insertions(+), 39 deletions(-)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 55a4ab26b348..f687df1edc60 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -211,7 +211,7 @@ typedef struct TransamVariablesData
/*
* These fields are protected by OidGenLock.
*/
- Oid nextOid; /* next OID to assign */
+ Oid8 nextOid; /* next OID (8 bytes) to assign */
uint32 oidCount; /* OIDs available before must do XLOG work */
/*
@@ -353,6 +353,7 @@ extern void SetTransactionIdLimit(TransactionId oldest_datfrozenxid,
extern void AdvanceOldestClogXid(TransactionId oldest_datfrozenxid);
extern bool ForceTransactionIdLimitUpdate(void);
extern Oid GetNewObjectId(void);
+extern Oid8 GetNewObjectId8(void);
extern void StopGeneratingPinnedObjectIds(void);
#ifdef USE_ASSERT_CHECKING
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 130ba9291096..7a590b7e1eaa 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -284,7 +284,7 @@ extern void ShutdownXLOG(int code, Datum arg);
extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
-extern void XLogPutNextOid(Oid nextOid);
+extern void XLogPutNextOid(Oid8 nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern XLogRecPtr XLogAssignLSN(void);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 7b5404460ec2..b95452ea41a5 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -43,7 +43,7 @@ typedef struct CheckPoint
int wal_level; /* current wal_level */
bool logicalDecodingEnabled; /* current logical decoding status */
FullTransactionId nextXid; /* next free transaction ID */
- Oid nextOid; /* next free OID */
+ Oid8 nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 2468a7d25785..64e749c1e6b7 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -101,7 +101,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
CheckPoint *checkpoint = (CheckPoint *) rec;
appendStringInfo(buf, "redo %X/%08X; "
- "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "
+ "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid " OID8_FORMAT "; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
@@ -129,10 +129,10 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
else if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
- memcpy(&nextOid, rec, sizeof(Oid));
- appendStringInfo(buf, "%u", nextOid);
+ memcpy(&nextOid, rec, sizeof(Oid8));
+ appendStringInfo(buf, OID8_FORMAT, nextOid);
}
else if (info == XLOG_RESTORE_POINT)
{
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index cb68d8fa974a..03209bc18c18 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -541,31 +541,51 @@ ForceTransactionIdLimitUpdate(void)
/*
- * GetNewObjectId -- allocate a new OID
+ * GetNewObjectId -- allocate a new OID (4 bytes)
*
- * OIDs are generated by a cluster-wide counter. Since they are only 32 bits
- * wide, counter wraparound will occur eventually, and therefore it is unwise
- * to assume they are unique unless precautions are taken to make them so.
- * Hence, this routine should generally not be used directly. The only direct
- * callers should be GetNewOidWithIndex() and GetNewRelFileNumber() in
- * catalog/catalog.c.
+ * OIDs are generated by a cluster-wide counter. The callers of this routine
+ * expect a 32 bit-wide counter, and counter wraparound will occur eventually,
+ * and therefore it is unwise to assume they are unique unless precautions are
+ * taken to make them so. This routine should generally not be used directly.
+ * The only direct callers should be GetNewOidWithIndex() and
+ * GetNewRelFileNumber() in catalog/catalog.c.
*/
Oid
GetNewObjectId(void)
{
- Oid result;
+ return (Oid) GetNewObjectId8();
+}
+
+/*
+ * GetNewObjectId8 -- allocate a new OID (8 bytes)
+ *
+ * This routine can be called directly if the consumer of the OID allocated
+ * stores the counter in an 8-byte space, where wraparound does not matter.
+ * We still need to care about the wraparound case in the low 32 bits of the
+ * space allocated, GetNewObjectId() expecting OIDs to never be allocated
+ * up to FirstNormalObjectId.
+ */
+Oid8
+GetNewObjectId8(void)
+{
+ Oid8 result;
+ Oid nextoid_lo;
+ uint32 nextoid_hi;
/* safety check, we should never get this far in a HS standby */
if (RecoveryInProgress())
elog(ERROR, "cannot assign OIDs during recovery");
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
+ nextoid_lo = (Oid) TransamVariables->nextOid;
+ nextoid_hi = (uint32) (TransamVariables->nextOid >> 32);
/*
- * Check for wraparound of the OID counter. We *must* not return 0
- * (InvalidOid), and in normal operation we mustn't return anything below
- * FirstNormalObjectId since that range is reserved for initdb (see
- * IsCatalogRelationOid()). Note we are relying on unsigned comparison.
+ * Check for wraparound of the OID counter in its lower 4 bytes. We
+ * *must* not return 0 (InvalidOid), and in normal operation we
+ * mustn't return anything below FirstNormalObjectId since that range
+ * is reserved for initdb (see IsCatalogRelationOid()). Note we are
+ * relying on unsigned comparison.
*
* During initdb, we start the OID generator at FirstGenbkiObjectId, so we
* only wrap if before that point when in bootstrap or standalone mode.
@@ -575,26 +595,32 @@ GetNewObjectId(void)
* available for automatic assignment during initdb, while ensuring they
* will never conflict with user-assigned OIDs.
*/
- if (TransamVariables->nextOid < ((Oid) FirstNormalObjectId))
+ if (nextoid_lo < ((Oid) FirstNormalObjectId))
{
if (IsPostmasterEnvironment)
{
/* wraparound, or first post-initdb assignment, in normal mode */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
else
{
/* we may be bootstrapping, so don't enforce the full range */
- if (TransamVariables->nextOid < ((Oid) FirstGenbkiObjectId))
+ if (nextoid_lo < ((Oid) FirstGenbkiObjectId))
{
/* wraparound in standalone mode (unlikely but possible) */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
}
}
+ /*
+ * Set next OID in its 8-byte space, skipping the first post-init
+ * assignment.
+ */
+ TransamVariables->nextOid = ((Oid8) nextoid_hi) << 32 | nextoid_lo;
+
/* If we run out of logged for use oids then we must log more */
if (TransamVariables->oidCount == 0)
{
@@ -619,7 +645,7 @@ GetNewObjectId(void)
* to the specified value.
*/
static void
-SetNextObjectId(Oid nextOid)
+SetNextObjectId(Oid8 nextOid)
{
/* Safety check, this is only allowable during initdb */
if (IsPostmasterEnvironment)
@@ -629,7 +655,7 @@ SetNextObjectId(Oid nextOid)
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
if (TransamVariables->nextOid > nextOid)
- elog(ERROR, "too late to advance OID counter to %u, it is now %u",
+ elog(ERROR, "too late to advance OID counter to " OID8_FORMAT ", it is now " OID8_FORMAT,
nextOid, TransamVariables->nextOid);
TransamVariables->nextOid = nextOid;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 3203f2fd4ee2..7c8acaecadb8 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8604,10 +8604,10 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
* Write a NEXTOID log record
*/
void
-XLogPutNextOid(Oid nextOid)
+XLogPutNextOid(Oid8 nextOid)
{
XLogBeginInsert();
- XLogRegisterData(&nextOid, sizeof(Oid));
+ XLogRegisterData(&nextOid, sizeof(Oid8));
(void) XLogInsert(RM_XLOG_ID, XLOG_NEXTOID);
/*
@@ -8873,7 +8873,7 @@ xlog_redo(XLogReaderState *record)
if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
/*
* We used to try to take the maximum of TransamVariables->nextOid and
@@ -8882,7 +8882,7 @@ xlog_redo(XLogReaderState *record)
* anyway, better to just believe the record exactly. We still take
* OidGenLock while setting the variable, just in case.
*/
- memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid));
+ memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid8));
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
TransamVariables->nextOid = nextOid;
TransamVariables->oidCount = 0;
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index acac97e89d38..fff8d57ac61d 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -840,7 +840,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
LSN_FORMAT_ARGS(checkPoint.redo),
wasShutdown ? "true" : "false"));
ereport(DEBUG1,
- (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: %u",
+ (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: " OID8_FORMAT,
U64FromFullTransactionId(checkPoint.nextXid),
checkPoint.nextOid)));
ereport(DEBUG1,
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 6fc87ed114d8..b785f7f40701 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -269,7 +269,7 @@ main(int argc, char *argv[])
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile->checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile->checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 79f3085d7696..68a176a2c427 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -82,7 +82,7 @@ static TransactionId oldest_commit_ts_xid_val;
static TransactionId newest_commit_ts_xid_val;
static bool next_oid_given = false;
-static Oid next_oid_val;
+static Oid8 next_oid_val;
static bool mxids_given = false;
static MultiXactId next_mxid_val;
@@ -253,7 +253,7 @@ main(int argc, char *argv[])
case 'o':
errno = 0;
- next_oid_val = strtouint32_strict(optarg, &endptr, 0);
+ next_oid_val = strtou64(optarg, &endptr, 0);
if (endptr == optarg || *endptr != '\0' || errno != 0)
{
pg_log_error("invalid argument for option %s", "-o");
@@ -775,7 +775,7 @@ PrintControlValues(bool guessed)
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile.checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile.checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile.checkPointCopy.nextMulti);
@@ -861,7 +861,7 @@ PrintNewControlValues(void)
if (next_oid_given)
{
- printf(_("NextOID: %u\n"),
+ printf(_("NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
}
@@ -1236,7 +1236,7 @@ usage(void)
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
- printf(_(" -o, --next-oid=OID set next OID\n"));
+ printf(_(" -o, --next-oid=OID8 set next OID (8 bytes)\n"));
printf(_(" -O, --multixact-offset=OFFSET set next multitransaction offset\n"));
printf(_(" -u, --oldest-transaction-id=XID set oldest transaction ID\n"));
printf(_(" -x, --next-transaction-id=XID set next transaction ID\n"));
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index 664c525f481c..a9580f35de95 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -286,11 +286,11 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>-o <replaceable class="parameter">oid</replaceable></option></term>
- <term><option>--next-oid=<replaceable class="parameter">oid</replaceable></option></term>
+ <term><option>-o <replaceable class="parameter">oid8</replaceable></option></term>
+ <term><option>--next-oid=<replaceable class="parameter">oid8</replaceable></option></term>
<listitem>
<para>
- Manually set the next OID.
+ Manually set the next OID (8 bytes).
</para>
<para>
--
2.55.0
v14-0005-Add-relation-option-toast_value_type.patch (text/plain, 6.4K) download
From c2a5993bab119d6a6012ac93fa1b8296b616f255 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:54:58 +0900
Subject: [PATCH v14 5/9] Add relation option toast_value_type
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is created
initially. This parameter has no effect if a TOAST table exists, even
after it is modified later on, even on rewrites.
This can be set only to "oid" currently, and will be expanded with a
second mode later.
Note: perhaps it would make sense to introduce that only when support
for 8-byte OID values are added to TOAST, the split is here to ease
review.
---
src/include/utils/rel.h | 17 +++++++++++++++++
src/backend/access/common/reloptions.c | 21 +++++++++++++++++++++
src/backend/catalog/toasting.c | 6 ++++++
src/bin/psql/tab-complete.in.c | 1 +
doc/src/sgml/ref/create_table.sgml | 18 ++++++++++++++++++
5 files changed, 63 insertions(+)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 41ab4586c6b5..f9e26ce6654a 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -341,11 +341,20 @@ typedef enum StdRdOptIndexCleanup
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
} StdRdOptIndexCleanup;
+/* StdRdOptions->toast_value_type values */
+typedef enum StdRdOptToastValueType
+{
+ STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+} StdRdOptToastValueType;
+
typedef struct StdRdOptions
{
int32 vl_len_; /* varlena header (do not touch directly!) */
int fillfactor; /* page fill factor in percent (0..100) */
int toast_tuple_target; /* target for tuple toasting */
+ StdRdOptToastValueType toast_value_type; /* type assigned to chunk_id
+ * at toast table creation */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
int parallel_workers; /* max number of parallel workers */
@@ -370,6 +379,14 @@ typedef struct StdRdOptions
((relation)->rd_options ? \
((StdRdOptions *) (relation)->rd_options)->toast_tuple_target : (defaulttarg))
+/*
+ * RelationGetToastValueType
+ * Returns the relation's toast_value_type. Note multiple eval of argument!
+ */
+#define RelationGetToastValueType(relation, defaulttarg) \
+ ((relation)->rd_options ? \
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)
+
/*
* RelationGetFillFactor
* Returns the relation's fillfactor. Note multiple eval of argument!
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index c3f6f45f4609..82491a31a9a4 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -548,6 +548,14 @@ static relopt_enum_elt_def viewCheckOptValues[] =
{(const char *) NULL} /* list terminator */
};
+/* values from StdRdOptToastValueType */
+static relopt_enum_elt_def StdRdOptToastValueTypes[] =
+{
+ /* no value for INVALID */
+ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {(const char *) NULL} /* list terminator */
+};
+
static relopt_enum enumRelOpts[] =
{
{
@@ -561,6 +569,17 @@ static relopt_enum enumRelOpts[] =
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")
},
+ {
+ {
+ "toast_value_type",
+ "Controls the attribute type of chunk_id at toast table creation",
+ RELOPT_KIND_HEAP,
+ ShareUpdateExclusiveLock
+ },
+ StdRdOptToastValueTypes,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ gettext_noop("Valid values are \"oid\".")
+ },
{
{
"buffering",
@@ -2078,6 +2097,8 @@ static const relopt_parse_elt stdRdOptionsTab[] = {
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_analyze_min_duration)},
{"toast_tuple_target", RELOPT_TYPE_INT,
offsetof(StdRdOptions, toast_tuple_target)},
+ {"toast_value_type", RELOPT_TYPE_ENUM,
+ offsetof(StdRdOptions, toast_value_type)},
{"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL,
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)},
{"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL,
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index bc0a72268335..28ddbd10d357 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -159,9 +159,15 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!IsBinaryUpgrade)
{
+ StdRdOptToastValueType value_type;
+
/* Normal mode, normal check */
if (!needs_toast_table(rel))
return false;
+
+ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
+ if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
+ toast_chunkid_typid = OIDOID;
}
else
{
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 1b74fa62c5c6..039639f07cb2 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -1450,6 +1450,7 @@ static const char *const table_storage_parameters[] = {
"toast.vacuum_max_eager_freeze_failure_rate",
"toast.vacuum_truncate",
"toast_tuple_target",
+ "toast_value_type",
"user_catalog_table",
"vacuum_index_cleanup",
"vacuum_max_eager_freeze_failure_rate",
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index e342585c7f08..3da754045c04 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1653,6 +1653,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry id="reloption-toast-value-type" xreflabel="toast_value_type">
+ <term><literal>toast_value_type</literal> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>toast_value_type</varname> storage parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ The toast_value_type specifies the attribute type of
+ <literal>chunk_id</literal> used when initially creating a toast
+ relation for this table.
+ By default this parameter is <literal>oid</literal>, to assign
+ <type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter cannot be set for TOAST tables.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="reloption-parallel-workers" xreflabel="parallel_workers">
<term><literal>parallel_workers</literal> (<type>integer</type>)
<indexterm>
--
2.55.0
v14-0006-Add-support-for-oid8-TOAST-values.patch (text/plain, 17.1K) download
From fa81c7899cc67614aceef1bdff4776b2a2d02088 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 6 Aug 2026 13:08:37 +0900
Subject: [PATCH v14 6/9] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, casted to OID (for now).
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 1 +
src/backend/access/common/toast_internals.c | 94 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 19 ++++-
src/backend/catalog/toasting.c | 24 +++++-
doc/src/sgml/ref/create_table.sgml | 2 +
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 22 +++--
9 files changed, 132 insertions(+), 41 deletions(-)
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f9e26ce6654a..b7da03446a94 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 82491a31a9a4..dfb28e592915 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..2c180491c2ff 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,8 +132,10 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -200,24 +203,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OIDOID)
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ Assert(false);
}
else
{
@@ -263,17 +274,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OIDOID)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
}
}
@@ -303,7 +319,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OIDOID)
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ else if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -384,6 +403,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -395,6 +415,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -405,10 +427,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +485,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +493,24 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..a93e98788da7 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +658,18 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 28ddbd10d357..66d483bb1be3 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -168,6 +169,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
toast_chunkid_typid = OIDOID;
+ else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)
+ toast_chunkid_typid = OID8OID;
}
else
{
@@ -200,7 +203,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -225,6 +229,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -343,7 +360,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OIDOID)
+ opclassIds[0] = OID_BTREE_OPS_OID;
+ else if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3da754045c04..3ed8f3581177 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1666,6 +1666,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
relation for this table.
By default this parameter is <literal>oid</literal>, to assign
<type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter can be set to <type>oid8</type> to use <type>oid8</type>
+ as attribute type for <literal>chunk_id</literal>.
This parameter cannot be set for TOAST tables.
</para>
</listitem>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 90aae3defcbc..21bd67c11ea0 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an OID8 identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 33b00fa4dc5c..42f409548e1d 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1876,8 +1876,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1885,10 +1889,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v14-0007-Add-tests-for-TOAST-relations-with-oid8-as-value.patch (text/plain, 22.9K) download
From 7fd2df7ee6b89bcbec97fc4bb430251745bf8dba Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 14:36:17 +0900
Subject: [PATCH v14 7/9] Add tests for TOAST relations with oid8 as value type
This adds coverage for relations created with default_toast_type =
'oid8', for external TOAST pointers both compressed and uncompressed.
---
src/test/regress/expected/strings.out | 254 ++++++++++++++++++----
src/test/regress/expected/type_sanity.out | 6 +-
src/test/regress/sql/strings.sql | 152 +++++++++----
src/test/regress/sql/type_sanity.sql | 6 +-
4 files changed, 337 insertions(+), 81 deletions(-)
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 313c50394650..2e46519df8cd 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2105,7 +2198,94 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index 1d21d3eb4467..4fabe08f74c0 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -578,15 +578,15 @@ WHERE c1.relnatts != (SELECT count(*) FROM pg_attribute AS a1
(0 rows)
-- Cross-check against pg_type entry
--- NOTE: we allow attstorage to be 'plain' even when typstorage is not;
--- this is mainly for toast tables.
+-- NOTE: we allow attstorage to be 'plain' or 'external' even when typstorage
+-- is not; this is mainly for toast tables.
SELECT a1.attrelid, a1.attname, t1.oid, t1.typname
FROM pg_attribute AS a1, pg_type AS t1
WHERE a1.atttypid = t1.oid AND
(a1.attlen != t1.typlen OR
a1.attalign != t1.typalign OR
a1.attbyval != t1.typbyval OR
- (a1.attstorage != t1.typstorage AND a1.attstorage != 'p'));
+ (a1.attstorage != t1.typstorage AND a1.attstorage NOT IN ('e', 'p')));
attrelid | attname | oid | typname
----------+---------+-----+---------
(0 rows)
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 38946e8954df..5e260ddaefc1 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,165 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
-- test internally compressing datums
diff --git a/src/test/regress/sql/type_sanity.sql b/src/test/regress/sql/type_sanity.sql
index 95d5b6e09151..5c166f00bb7e 100644
--- a/src/test/regress/sql/type_sanity.sql
+++ b/src/test/regress/sql/type_sanity.sql
@@ -420,8 +420,8 @@ WHERE c1.relnatts != (SELECT count(*) FROM pg_attribute AS a1
WHERE a1.attrelid = c1.oid AND a1.attnum > 0);
-- Cross-check against pg_type entry
--- NOTE: we allow attstorage to be 'plain' even when typstorage is not;
--- this is mainly for toast tables.
+-- NOTE: we allow attstorage to be 'plain' or 'external' even when typstorage
+-- is not; this is mainly for toast tables.
SELECT a1.attrelid, a1.attname, t1.oid, t1.typname
FROM pg_attribute AS a1, pg_type AS t1
@@ -429,7 +429,7 @@ WHERE a1.atttypid = t1.oid AND
(a1.attlen != t1.typlen OR
a1.attalign != t1.typalign OR
a1.attbyval != t1.typbyval OR
- (a1.attstorage != t1.typstorage AND a1.attstorage != 'p'));
+ (a1.attstorage != t1.typstorage AND a1.attstorage NOT IN ('e', 'p')));
-- Look for IsCatalogTextUniqueIndexOid() omissions.
--
2.55.0
v14-0008-Add-support-for-TOAST-pointers-as-oid8.patch (text/plain, 54.0K) download
From e0fd92d5d4dc1e3a7a26601cebaad97a45e154ac Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 16:34:22 +0900
Subject: [PATCH v14 8/9] Add support for TOAST pointers as oid8
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 3 +
src/include/access/heaptoast.h | 16 +-
src/include/varatt.h | 82 ++++-
src/backend/access/common/detoast.c | 171 +++++++---
src/backend/access/common/reloptions.c | 2 +-
src/backend/access/common/toast_compression.c | 11 +-
src/backend/access/common/toast_internals.c | 314 ++++++++++++------
src/backend/access/heap/heaptoast.c | 5 +-
src/backend/access/table/toast_helper.c | 4 +
.../replication/logical/reorderbuffer.c | 44 ++-
src/backend/utils/adt/varlena.c | 19 +-
src/test/regress/expected/compression.out | 41 +++
src/test/regress/expected/strings.out | 43 +++
src/test/regress/sql/compression.sql | 22 ++
src/test/regress/sql/strings.sql | 25 ++
doc/src/sgml/storage.sgml | 10 +-
contrib/amcheck/verify_heapam.c | 103 ++++--
17 files changed, 702 insertions(+), 213 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..f065ea034e98 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -30,6 +30,9 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..b7ef69a8f56b 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,13 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 33979bbaaad1..bb1a59bf6cc9 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -38,6 +38,38 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except that va_valueid is split into two uint32 halves to enforce a
+ * stricter structure size without padding.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==
+ sizeof(varatt_external_oid8),
+ "varatt_external_oid8 should have no padding");
+
+/* Helper macros to get/set the 64-bit value ID from the lo/hi fields */
+#define VARATT_EXTERNAL_OID8_GET_VALUEID(tp) \
+ ((Oid8) (tp).va_valueid_lo | ((Oid8) (tp).va_valueid_hi << 32))
+#define VARATT_EXTERNAL_OID8_SET_VALUEID(tp, id) \
+ do { \
+ (tp).va_valueid_lo = (uint32) (id); \
+ (tp).va_valueid_hi = (uint32) ((id) >> 32); \
+ } while (0)
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -87,6 +119,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -108,6 +141,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -361,7 +396,19 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
+}
+
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
+{
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
}
/* Is varlena datum an indirect pointer? */
@@ -519,16 +566,6 @@ VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
* Testing whether an externally-stored value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -543,4 +580,27 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid8.
+ */
+static inline Size
+VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+}
+
+static inline uint32
+VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+}
+
+static inline bool
+VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)
+{
+ return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <
+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+}
+
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..e1919809751c 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,40 +225,51 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /* Extract extsize and compression info from the appropriate pointer */
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
- * For compressed values, we need to fetch enough slices to decompress
- * at least the requested part (when a prefix is requested).
- * Otherwise, just fetch all slices.
+ * For compressed values, we need to fetch enough slices to
+ * decompress at least the requested part (when a prefix is
+ * requested). Otherwise, just fetch all slices.
+ *
+ * At least for now, if it's LZ4 data, we'll have to fetch the
+ * whole thing, because there doesn't seem to be an API call to
+ * determine how much compressed data we need to be sure of being
+ * able to decompress the required slice.
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
- /*
- * Determine maximum amount of compressed data needed for a prefix
- * of a given length (after decompression).
- *
- * At least for now, if it's LZ4 data, we'll have to fetch the
- * whole thing, because there doesn't seem to be an API call to
- * determine how much compressed data we need to be sure of being
- * able to decompress the required slice.
- */
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,23 +355,47 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- result = (varlena *) palloc(attrsize + VARHDRSZ);
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
+
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -369,10 +404,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +433,43 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +482,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +495,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +504,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +605,20 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +675,20 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index dfb28e592915..ea9a04179090 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -579,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")
+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 3b1bd6519261..53b31943cdfe 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -260,7 +260,16 @@ toast_get_compression_id(varlena *attr)
* the external toast pointer. If compressed inline, fetch it from the
* toast compression header.
*/
- if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ cmid = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ }
+ else if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
varatt_external_oid toast_pointer;
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 2c180491c2ff..7b37798e1c08 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -125,7 +125,6 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -133,9 +132,17 @@ toast_save_datum(Relation rel, Datum value,
int num_indexes;
int validIndex;
Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
Assert(OidIsValid(toast_typid));
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -165,28 +172,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -198,13 +209,14 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or OID8 TOAST relations.
+ * Choose a new value to use as the value ID for this toast value, and
+ * determine the maximum chunk size based on the TOAST table's chunk_id
+ * type.
*
* Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
@@ -213,84 +225,123 @@ toast_save_datum(Relation rel, Datum value,
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * conflict with either new or existing toast value IDs. For Oid8 tables,
+ * value conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OIDOID)
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
+ if (!OidIsValid(rel->rd_toastoid))
+ {
+ /* normal case: just choose a new Oid8 */
+ va_valueid = GetNewObjectId8();
+ }
else
- Assert(false);
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid8;
+ if (oldexternal != NULL)
+ {
+ varatt_external_oid8 old_toast_pointer;
+
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /* This value came from the old toast table; reuse its ID */
+ va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);
+ /*
+ * There is a corner case here: the table rewrite might
+ * have to copy both live and recently-dead versions of a
+ * row, and those versions could easily reference the same
+ * toast value. When we copy the second or later version
+ * of such a row, reusing the OID will mean we select an
+ * OID that's already in the new toast table. Check for
+ * that, and if so, just fall through without writing the
+ * data again.
+ *
+ * While annoying and ugly-looking, this is a good thing
+ * because it ensures that we wind up with only one copy
+ * of the toast value when there is only one copy in the
+ * old toast table. Before we detected this case, we'd
+ * have made multiple copies, wasting space; and what's
+ * worse, the copies belonging to already-deleted heap
+ * tuples would not be reclaimed by VACUUM.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
+ }
+ }
+ if (va_valueid == InvalidOid8)
+ {
+ /*
+ * New value. For Oid8 we just pick a new ID without worrying
+ * about conflicts.
+ */
+ va_valueid = GetNewObjectId8();
+ }
+ }
+
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
+ if (!OidIsValid(rel->rd_toastoid))
{
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ /* normal case: just choose an unused OID */
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
+ else
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid;
+ if (oldexternal != NULL)
{
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
+ varatt_external_oid old_toast_pointer;
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
{
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
+ /* This value came from the old toast table; reuse its OID */
+ va_valueid = old_toast_pointer.va_valueid;
+
+ /*
+ * Corner case during table rewrite with multiple
+ * versions of the same row. See above for details.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /* Match, so short-circuit the data storage loop below */
+ data_todo = 0;
+ }
}
}
- }
- if (toast_pointer.va_valueid == InvalidOid)
- {
- /*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
- */
- if (toast_typid == OIDOID)
+ if (va_valueid == InvalidOid)
{
+ /*
+ * new value; must choose a value that doesn't conflict in
+ * either old or new toast table.
+ */
do
{
- toast_pointer.va_valueid =
+ va_valueid =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ va_valueid));
}
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
}
+
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
/*
@@ -314,15 +365,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
- if (toast_typid == OIDOID)
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
- else if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -376,11 +427,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -395,7 +469,6 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -403,42 +476,75 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toast_typid;
+ Oid toastrelid;
+ vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
/*
- * Open the toast relation and its indexes
+ * Determine the pointer type from the datum's vartag and extract the
+ * toast relation OID and value ID accordingly. The vartag tells us
+ * everything we need - no TOAST table schema lookup required.
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
- toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ tag = VARTAG_EXTERNAL(attr);
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
+ if (tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- if (toast_typid == OIDOID)
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toastrelid = toast_pointer8.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ Assert(tag == VARTAG_ONDISK_OID);
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toastrelid = toast_pointer.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(toast_pointer.va_valueid));
- else
- Assert(false);
+ }
/*
* Find all the chunks. (We don't actually care whether we see them in
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index a93e98788da7..364aec11fc88 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -650,7 +650,10 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ if (toast_typid == OID8OID)
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+ else
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6f2d691cc681..6c7257f5f94b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc)
/*
* If the old value is stored on disk, check if it has changed so
* we have to delete it later.
+ *
+ * Note that TOAST pointers could have different vartags, for oid
+ * or oid8, and these can have a different sizes.
*/
if (att->attlen == -1 && !ttc->ttc_oldisnull[i] &&
VARATT_IS_EXTERNAL_ONDISK(old_value))
{
if (ttc->ttc_isnull[i] ||
!VARATT_IS_EXTERNAL_ONDISK(new_value) ||
+ VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) ||
memcmp(old_value, new_value,
VARSIZE_EXTERNAL(old_value)) != 0)
{
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dd4b6e8f7f45..644e1415cec2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5176,12 +5176,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* va_rawsize is the size of the original datum -- including header */
varatt_external_oid toast_pointer;
+ varatt_external_oid8 toast_pointer8;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
+ int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5201,8 +5203,19 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (!VARATT_IS_EXTERNAL(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ /* Branch on vartag to handle both pointer types */
+ if (VARTAG_EXTERNAL(varlena_pointer) == VARTAG_ONDISK_OID8)
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);
+ toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ rawsize = toast_pointer8.va_rawsize;
+ }
+ else
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
+ rawsize = toast_pointer.va_rawsize;
+ }
/*
* Check whether the toast tuple changed, replace if so.
@@ -5220,7 +5233,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(rawsize);
ent->reconstructed = reconstructed;
@@ -5245,13 +5258,28 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ /* Verify size and set compression status based on pointer type */
+ if (VARTAG_EXTERNAL(varlena_pointer) == VARTAG_ONDISK_OID8)
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 1e264904a6e1..e104a6308d65 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,7 +4261,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external_oid toast_pointer;
+ Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4288,9 +4288,22 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ {
+ varatt_external_oid8 toast_pointer;
- PG_RETURN_OID8(toast_pointer.va_valueid);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_valueid;
+ }
+
+ PG_RETURN_OID8(result);
}
/*
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 09f198149aa4..e9128193515a 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
(1 row)
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+ uncompressed
+--------------
+ t
+(1 row)
+
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+ uncompressed
+--------------
+ t
+(1 row)
+
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+DROP TABLE toastcomp_oid, toastcomp_oid8;
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 2e46519df8cd..ba1961baa155 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -2286,6 +2286,49 @@ SELECT c1.relname, a.atttypid::regtype
ALTER TABLE toasttest_oid RESET (toast_value_type);
ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index ce5ea37a660c..229143f06ddd 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2;
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+DROP TABLE toastcomp_oid, toastcomp_oid8;
+
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 5e260ddaefc1..5c41fb2d1ea5 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -723,6 +723,31 @@ SELECT c1.relname, a.atttypid::regtype
ALTER TABLE toasttest_oid RESET (toast_value_type);
ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+
+DROP TABLE toastupd_oid, toastupd_oid8;
+
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 21bd67c11ea0..3561a6a67ece 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,9 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the
+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 42f409548e1d..ef631408b8ef 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1566,9 +1568,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 max_chunk_size;
Oid8 toast_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ toast_valueid = ta->va_valueid;
+
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1672,8 +1677,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1732,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1777,26 +1784,46 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
/*
- * Must copy attr into toast_pointer for alignment considerations
+ * Must copy attr into toast_pointer for alignment considerations.
+ * Branch on the tag to determine which pointer type to extract.
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ va_tag_value = VARTAG_EXTERNAL(attr);
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ va_rawsize = toast_pointer8.va_rawsize;
+ va_extinfo = toast_pointer8.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
+ va_rawsize = toast_pointer.va_rawsize;
+ va_extinfo = toast_pointer.va_extinfo;
+ }
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1850,7 +1877,23 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ ta->tag = va_tag_value;
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 tp;
+
+ VARATT_EXTERNAL_GET_POINTER(tp, attr);
+ ta->va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(tp);
+ ta->va_extinfo = tp.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid tp;
+
+ VARATT_EXTERNAL_GET_POINTER(tp, attr);
+ ta->va_valueid = tp.va_valueid;
+ ta->va_extinfo = tp.va_extinfo;
+ }
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1878,29 +1921,33 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
int32 max_chunk_size;
Oid8 toast_valueid;
- Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
+ if (ta->tag == VARTAG_ONDISK_OID8)
+ {
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ ObjectId8GetDatum(toast_valueid));
+ }
else
- Assert(false);
+ {
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) toast_valueid));
+ }
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1920,8 +1967,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
--
2.55.0
v14-0009-Fix-toast_tuple_find_biggest_attribute-for-OID8-.patch (text/plain, 6.7K) download
From c2a987dde5297b63d2ddb71c7ab473135f9e9b8b Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 15:32:34 +0900
Subject: [PATCH v14 9/9] Fix toast_tuple_find_biggest_attribute() for OID8
TOAST values
This routine is used to decide if largest varlena attributes should be
compressed or not, but stuck its decisions to a hardcoded
TOAST_OID_POINTER_SIZE, which would be amiss when dealing with an oid8
TOAST value.
This information is added to the parent Relation, and can be asked on a
per-call basis of a new RelationGetToastChunkIdType().
---
src/include/utils/rel.h | 9 +++++
src/include/utils/relcache.h | 1 +
src/backend/access/common/toast_internals.c | 3 +-
src/backend/access/table/toast_helper.c | 18 ++++++++--
src/backend/utils/cache/relcache.c | 37 +++++++++++++++++++++
5 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b7da03446a94..280a9b3ea21b 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -250,6 +250,15 @@ typedef struct RelationData
*/
Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */
+ /*
+ * Type OID of the "chunk_id" column of this relation's TOAST table, i.e.
+ * OIDOID or OID8OID.
+ *
+ * This data is filled on demand by RelationGetToastChunkIdType(), not
+ * at relcache build time, so as to save in syscache lookups.
+ */
+ Oid rd_toastchunkidtype;
+
bool pgstat_enabled; /* should relation stats be counted */
/* use "struct" here to avoid needing to include pgstat.h: */
struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 89c27aa1529f..e17c84908332 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok);
extern Oid RelationGetReplicaIndex(Relation relation);
+extern Oid RelationGetToastChunkIdType(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
extern List *RelationGetDummyIndexExpressions(Relation relation);
extern List *RelationGetIndexPredicate(Relation relation);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 7b37798e1c08..465f0d1e098f 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,7 +25,6 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,7 +130,7 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
- Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ Oid toast_typid = RelationGetToastChunkIdType(rel);
int32 max_chunk_size;
/* Fields that will be assembled into the TOAST pointer at the end */
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6c7257f5f94b..3235d033ea9b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -175,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
- * if not, no benefit is to be expected by compressing it.
+ * The column must be larger than the TOAST pointer that would replace it;
+ * if not, no benefit is to be expected by compressing it. Note that this
+ * choice depends on the TOAST value type, oid or oid8.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -188,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+ int32 biggest_size;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /*
+ * Size of the TOAST pointer this relation would use. A relation without
+ * a TOAST table cannot have any of its attributes moved out-of-line, but
+ * it can still have some of them compressed. Fall back to the oid size
+ * in that case.
+ */
+ if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID)
+ biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE);
+ else
+ biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index f475d7039772..65dfdbda100a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1205,6 +1205,9 @@ retry:
relation->rd_fkeylist = NIL;
relation->rd_fkeyvalid = false;
+ /* TOAST type data is not loaded till asked for */
+ relation->rd_toastchunkidtype = InvalidOid;
+
/* partitioning data is not loaded till asked for */
relation->rd_partkey = NULL;
relation->rd_partkeycxt = NULL;
@@ -5093,6 +5096,39 @@ RelationGetReplicaIndex(Relation relation)
return relation->rd_replidindex;
}
+/*
+ * RelationGetToastChunkIdType -- get the type of the relation's TOAST
+ * table "chunk_id" column
+ *
+ * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST
+ * table.
+ */
+Oid
+RelationGetToastChunkIdType(Relation relation)
+{
+ Oid toastrelid = relation->rd_rel->reltoastrelid;
+ Oid typid;
+
+ /* Quick exit if we already computed the value */
+ if (OidIsValid(relation->rd_toastchunkidtype))
+ return relation->rd_toastchunkidtype;
+
+ /* Nothing to report without a TOAST table */
+ if (!OidIsValid(toastrelid))
+ return InvalidOid;
+
+ typid = get_atttype(toastrelid, 1);
+ if (!OidIsValid(typid))
+ elog(ERROR, "cache lookup failed for TOAST relation %u",
+ toastrelid);
+ if (typid != OIDOID && typid != OID8OID)
+ elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u",
+ typid, toastrelid);
+
+ relation->rd_toastchunkidtype = typid;
+ return typid;
+}
+
/*
* RelationGetIndexExpressions -- get the index expressions for an index
*
@@ -6530,6 +6566,7 @@ load_relcache_init_file(bool shared)
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
rel->rd_droppedSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_toastchunkidtype = InvalidOid;
rel->pgstat_info = NULL;
/*
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-08 03:50 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-08 03:50 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 07, 2026 at 04:54:35PM +0900, Michael Paquier wrote:
> Attached is v14, with all that addressed (should perhaps have moved
> the tests in 0008 to 0007, but it's late here and that does not change
> the stuff to review).
While doing a new round of self-review, that included some tests with
pg_upgrade, I have spotted a hole in the control file: the nextOid was
stored as an Oid instead of an Oid8. This would be bad when
upgrading, because we would conflict for oid8 TOAST values once the
counter gets higher than 2^32. This has also spotted a gap in
pg_resetwal and pg_control_checkpoint(), where we need to show the new
information as an oid8.
Thinking more about which part of the patch set makes the most sense
as an independent piece, I have decided to split the upgrade of
nextOid to Oid8 in the control file and the introduction of the new
GetNewObjectId8() as a patch of their own, labelled now at 0001. This
includes a battery of tests for pg_upgrade to check that the value is
correctly carried, resetwal and pg_control_checkpoint(). As a whole,
I'm feeling pretty much OK with how this part bundles.
Small note: while hacking on 0001 I have managed to break pg_resetwal
-o with negative values. Added a test for that on HEAD separately,
while on it..
There is still something that I am wondering about this part of the
patch, though, should we still publish in the output of
pg_control_checkpoint() and pg_controldata an Oid extracted from the
Oid8 value? I have not done that, and that does not make the control
file larger. Just wondering if that would be useful to keep because
we cannot really do an oid8->oid cast, and the oid information could
still be useful on its own?
The rest of the patch set is mostly following the same organization,
and I have spent a few hours working more on the last piece, making
cleaner the contract with the vartag handling (one could not the
changes in varatt.h, where I have changed to inline function for the
get/set Oid8 values in varatt_external_oid8).
0005~0007 in the set of patches cover gaps in the regression tests.
0005 is the same one set of SQLs as the previous patch set (mostly),
0006 and 0007 cover more cases with test_decoding, amcheck,
reloptions, the compression function, and the external toast pointer
comparison in toast_tuple_init(). They have been split as they can
still work once the reloption supports oid8, even if we don't make the
varatt_external added yet (last patch).
--
Michael
Attachments: download all (12 files, .tar.gz)
v15-0001-Enlarge-OID-generation-to-8-bytes.patch (text/plain, 23.3K) download
From 7e71e23982b4fe1ff105b4e7f7731b5c10bdb0af Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 12:44:07 +0900
Subject: [PATCH v15 01/11] Enlarge OID generation to 8 bytes
This change is combined with an update of the control file, with
CheckPoint.nextOid enlarged from Oid to Oid8. pg_control_checkpoint(),
pg_resetwal and pg_upgrade are updated to accomodate to this change, to
make sure that the value is not truncated due to 32-bit limitations when
used.
This commit adds a new routine called GetNewObjectId8() in varsup.c,
which is able to retrieve a 8-byte OID. GetNewObjectId() is kept
compatible with its origin, where we still check that the lower 32 bits
of the counter do not wraparound, handling the FirstNormalObjectId case.
A couple of comments in the xlog redo logic referred to wraparound
issues, which is not the case anymore with this change in place, so
refresh these.
Tests are added to cover that, across all the tools covered by this
change.
Bump catalog version.
Bump XLOG_PAGE_MAGIC, due to XLOG_NEXTOID.
---
src/include/access/transam.h | 3 +-
src/include/access/xlog.h | 2 +-
src/include/access/xlog_internal.h | 2 +-
src/include/catalog/pg_control.h | 4 +-
src/include/catalog/pg_proc.dat | 2 +-
src/backend/access/rmgrdesc/xlogdesc.c | 8 +--
src/backend/access/transam/varsup.c | 59 ++++++++++++++++-------
src/backend/access/transam/xlog.c | 27 ++++++-----
src/backend/access/transam/xlogrecovery.c | 2 +-
src/backend/utils/misc/pg_controldata.c | 2 +-
src/bin/pg_controldata/pg_controldata.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 10 ++--
src/bin/pg_resetwal/t/001_basic.pl | 17 +++++++
src/bin/pg_upgrade/controldata.c | 2 +-
src/bin/pg_upgrade/pg_upgrade.c | 2 +-
src/bin/pg_upgrade/pg_upgrade.h | 2 +-
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 25 ++++++++++
doc/src/sgml/func/func-info.sgml | 2 +-
doc/src/sgml/ref/pg_resetwal.sgml | 6 +--
19 files changed, 123 insertions(+), 56 deletions(-)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 55a4ab26b348..f687df1edc60 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -211,7 +211,7 @@ typedef struct TransamVariablesData
/*
* These fields are protected by OidGenLock.
*/
- Oid nextOid; /* next OID to assign */
+ Oid8 nextOid; /* next OID (8 bytes) to assign */
uint32 oidCount; /* OIDs available before must do XLOG work */
/*
@@ -353,6 +353,7 @@ extern void SetTransactionIdLimit(TransactionId oldest_datfrozenxid,
extern void AdvanceOldestClogXid(TransactionId oldest_datfrozenxid);
extern bool ForceTransactionIdLimitUpdate(void);
extern Oid GetNewObjectId(void);
+extern Oid8 GetNewObjectId8(void);
extern void StopGeneratingPinnedObjectIds(void);
#ifdef USE_ASSERT_CHECKING
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 130ba9291096..7a590b7e1eaa 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -284,7 +284,7 @@ extern void ShutdownXLOG(int code, Datum arg);
extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
-extern void XLogPutNextOid(Oid nextOid);
+extern void XLogPutNextOid(Oid8 nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern XLogRecPtr XLogAssignLSN(void);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index be718993401b..bf609c3c703d 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -32,7 +32,7 @@
/*
* Each page of XLOG file has a header like this:
*/
-#define XLOG_PAGE_MAGIC 0xD121 /* can be used as WAL version indicator */
+#define XLOG_PAGE_MAGIC 0xD122 /* can be used as WAL version indicator */
typedef struct XLogPageHeaderData
{
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 7b5404460ec2..89ab43dd4fc7 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -22,7 +22,7 @@
/* Version identifier for this pg_control format */
-#define PG_CONTROL_VERSION 1903
+#define PG_CONTROL_VERSION 2000
/* Nonce key length, see below */
#define MOCK_AUTH_NONCE_LEN 32
@@ -43,7 +43,7 @@ typedef struct CheckPoint
int wal_level; /* current wal_level */
bool logicalDecodingEnabled; /* current logical decoding status */
FullTransactionId nextXid; /* next free transaction ID */
- Oid nextOid; /* next free OID */
+ Oid8 nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 960763ee50ba..c53ce68c7173 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12429,7 +12429,7 @@
descr => 'pg_controldata checkpoint state information as a function',
proname => 'pg_control_checkpoint', provolatile => 'v',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,bool,text,oid,xid,int8,xid,oid,xid,xid,oid,xid,xid,int4,timestamptz}',
+ proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,bool,text,oid8,xid,int8,xid,oid,xid,xid,oid,xid,xid,int4,timestamptz}',
proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,logical_decoding,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,data_page_checksum_version,checkpoint_time}',
prosrc => 'pg_control_checkpoint' },
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 2468a7d25785..64e749c1e6b7 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -101,7 +101,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
CheckPoint *checkpoint = (CheckPoint *) rec;
appendStringInfo(buf, "redo %X/%08X; "
- "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "
+ "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid " OID8_FORMAT "; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
@@ -129,10 +129,10 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
else if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
- memcpy(&nextOid, rec, sizeof(Oid));
- appendStringInfo(buf, "%u", nextOid);
+ memcpy(&nextOid, rec, sizeof(Oid8));
+ appendStringInfo(buf, OID8_FORMAT, nextOid);
}
else if (info == XLOG_RESTORE_POINT)
{
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index cb68d8fa974a..889bdebe1880 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -541,31 +541,51 @@ ForceTransactionIdLimitUpdate(void)
/*
- * GetNewObjectId -- allocate a new OID
+ * GetNewObjectId -- allocate a new OID (4 bytes)
*
- * OIDs are generated by a cluster-wide counter. Since they are only 32 bits
- * wide, counter wraparound will occur eventually, and therefore it is unwise
- * to assume they are unique unless precautions are taken to make them so.
- * Hence, this routine should generally not be used directly. The only direct
- * callers should be GetNewOidWithIndex() and GetNewRelFileNumber() in
- * catalog/catalog.c.
+ * OIDs are generated by a cluster-wide counter. The callers of this routine
+ * expect a 32 bit-wide counter. Counter wraparound will occur eventually,
+ * and therefore it is unwise to assume they are unique unless precautions are
+ * taken to make them so. This routine should generally not be used directly.
+ * The only direct callers should be GetNewOidWithIndex() and
+ * GetNewRelFileNumber() in catalog/catalog.c.
*/
Oid
GetNewObjectId(void)
{
- Oid result;
+ return (Oid) GetNewObjectId8();
+}
+
+/*
+ * GetNewObjectId8 -- allocate a new OID (8 bytes)
+ *
+ * This routine can be called directly if the consumer of the OID allocated
+ * stores the counter in an 8-byte space, where wraparound does not matter.
+ * We still need to care about the wraparound case in the low 32 bits of the
+ * space allocated, GetNewObjectId() expecting OIDs to never be allocated
+ * up to FirstNormalObjectId.
+ */
+Oid8
+GetNewObjectId8(void)
+{
+ Oid8 result;
+ Oid nextoid_lo;
+ uint32 nextoid_hi;
/* safety check, we should never get this far in a HS standby */
if (RecoveryInProgress())
elog(ERROR, "cannot assign OIDs during recovery");
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
+ nextoid_lo = (Oid) TransamVariables->nextOid;
+ nextoid_hi = (uint32) (TransamVariables->nextOid >> 32);
/*
- * Check for wraparound of the OID counter. We *must* not return 0
- * (InvalidOid), and in normal operation we mustn't return anything below
- * FirstNormalObjectId since that range is reserved for initdb (see
- * IsCatalogRelationOid()). Note we are relying on unsigned comparison.
+ * Check for wraparound of the OID counter in its lower 4 bytes. We *must*
+ * not return 0 (InvalidOid), and in normal operation we mustn't return
+ * anything below FirstNormalObjectId since that range is reserved for
+ * initdb (see IsCatalogRelationOid()). Note we are relying on unsigned
+ * comparison.
*
* During initdb, we start the OID generator at FirstGenbkiObjectId, so we
* only wrap if before that point when in bootstrap or standalone mode.
@@ -575,26 +595,29 @@ GetNewObjectId(void)
* available for automatic assignment during initdb, while ensuring they
* will never conflict with user-assigned OIDs.
*/
- if (TransamVariables->nextOid < ((Oid) FirstNormalObjectId))
+ if (nextoid_lo < ((Oid) FirstNormalObjectId))
{
if (IsPostmasterEnvironment)
{
/* wraparound, or first post-initdb assignment, in normal mode */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
else
{
/* we may be bootstrapping, so don't enforce the full range */
- if (TransamVariables->nextOid < ((Oid) FirstGenbkiObjectId))
+ if (nextoid_lo < ((Oid) FirstGenbkiObjectId))
{
/* wraparound in standalone mode (unlikely but possible) */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
}
}
+ /* Set next OID in its 8-byte space */
+ TransamVariables->nextOid = ((Oid8) nextoid_hi) << 32 | nextoid_lo;
+
/* If we run out of logged for use oids then we must log more */
if (TransamVariables->oidCount == 0)
{
@@ -619,7 +642,7 @@ GetNewObjectId(void)
* to the specified value.
*/
static void
-SetNextObjectId(Oid nextOid)
+SetNextObjectId(Oid8 nextOid)
{
/* Safety check, this is only allowable during initdb */
if (IsPostmasterEnvironment)
@@ -629,7 +652,7 @@ SetNextObjectId(Oid nextOid)
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
if (TransamVariables->nextOid > nextOid)
- elog(ERROR, "too late to advance OID counter to %u, it is now %u",
+ elog(ERROR, "too late to advance OID counter to " OID8_FORMAT ", it is now " OID8_FORMAT,
nextOid, TransamVariables->nextOid);
TransamVariables->nextOid = nextOid;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 3203f2fd4ee2..b84eac793ce8 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8604,10 +8604,10 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
* Write a NEXTOID log record
*/
void
-XLogPutNextOid(Oid nextOid)
+XLogPutNextOid(Oid8 nextOid)
{
XLogBeginInsert();
- XLogRegisterData(&nextOid, sizeof(Oid));
+ XLogRegisterData(&nextOid, sizeof(Oid8));
(void) XLogInsert(RM_XLOG_ID, XLOG_NEXTOID);
/*
@@ -8873,16 +8873,17 @@ xlog_redo(XLogReaderState *record)
if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
/*
* We used to try to take the maximum of TransamVariables->nextOid and
- * the recorded nextOid, but that fails if the OID counter wraps
- * around. Since no OID allocation should be happening during replay
- * anyway, better to just believe the record exactly. We still take
- * OidGenLock while setting the variable, just in case.
+ * the recorded nextOid, but that failed back when the counter was 4
+ * bytes wide and could wrap around. Since no OID allocation should
+ * be happening during replay anyway, better to just believe the
+ * record exactly. We still take OidGenLock while setting the
+ * variable, just in case.
*/
- memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid));
+ memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid8));
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
TransamVariables->nextOid = nextOid;
TransamVariables->oidCount = 0;
@@ -9010,11 +9011,11 @@ xlog_redo(XLogReaderState *record)
* to track OID assignment through XLOG_NEXTOID records. The nextOid
* counter is from the start of the checkpoint and might well be stale
* compared to later XLOG_NEXTOID records. We could try to take the
- * maximum of the nextOid counter and our latest value, but since
- * there's no particular guarantee about the speed with which the OID
- * counter wraps around, that's a risky thing to do. In any case,
- * users of the nextOid counter are required to avoid assignment of
- * duplicates, so that a somewhat out-of-date value should be safe.
+ * maximum of the nextOid counter and our latest value, but due to the
+ * guarantees offered by XLogPutNextOid(), this is not necessary. In
+ * any case, users of the nextOid counter are required to avoid
+ * assignment of duplicates, so that a somewhat out-of-date value
+ * should be safe.
*/
/* Handle multixact */
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index acac97e89d38..fff8d57ac61d 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -840,7 +840,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
LSN_FORMAT_ARGS(checkPoint.redo),
wasShutdown ? "true" : "false"));
ereport(DEBUG1,
- (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: %u",
+ (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: " OID8_FORMAT,
U64FromFullTransactionId(checkPoint.nextXid),
checkPoint.nextOid)));
ereport(DEBUG1,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 9014f0953e99..1f47302fe2a7 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -124,7 +124,7 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid)));
nulls[7] = false;
- values[8] = ObjectIdGetDatum(ControlFile->checkPointCopy.nextOid);
+ values[8] = ObjectId8GetDatum(ControlFile->checkPointCopy.nextOid);
nulls[8] = false;
values[9] = TransactionIdGetDatum(ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 6fc87ed114d8..b785f7f40701 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -269,7 +269,7 @@ main(int argc, char *argv[])
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile->checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile->checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 79f3085d7696..63e4381e03f0 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -82,7 +82,7 @@ static TransactionId oldest_commit_ts_xid_val;
static TransactionId newest_commit_ts_xid_val;
static bool next_oid_given = false;
-static Oid next_oid_val;
+static Oid8 next_oid_val;
static bool mxids_given = false;
static MultiXactId next_mxid_val;
@@ -253,7 +253,7 @@ main(int argc, char *argv[])
case 'o':
errno = 0;
- next_oid_val = strtouint32_strict(optarg, &endptr, 0);
+ next_oid_val = strtouint64_strict(optarg, &endptr, 0);
if (endptr == optarg || *endptr != '\0' || errno != 0)
{
pg_log_error("invalid argument for option %s", "-o");
@@ -775,7 +775,7 @@ PrintControlValues(bool guessed)
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile.checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile.checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile.checkPointCopy.nextMulti);
@@ -861,7 +861,7 @@ PrintNewControlValues(void)
if (next_oid_given)
{
- printf(_("NextOID: %u\n"),
+ printf(_("NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
}
@@ -1236,7 +1236,7 @@ usage(void)
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
- printf(_(" -o, --next-oid=OID set next OID\n"));
+ printf(_(" -o, --next-oid=OID8 set next OID (8 bytes)\n"));
printf(_(" -O, --multixact-offset=OFFSET set next multitransaction offset\n"));
printf(_(" -u, --oldest-transaction-id=XID set oldest transaction ID\n"));
printf(_(" -x, --next-transaction-id=XID set next transaction ID\n"));
diff --git a/src/bin/pg_resetwal/t/001_basic.pl b/src/bin/pg_resetwal/t/001_basic.pl
index 6af97c0a7d99..f54e9bb42231 100644
--- a/src/bin/pg_resetwal/t/001_basic.pl
+++ b/src/bin/pg_resetwal/t/001_basic.pl
@@ -265,4 +265,21 @@ command_like(
$node->start;
ok(1, 'server started after reset');
+# Checks for 8 bytes OIDs.
+$node->stop;
+my $new_nextoid = "4295067296"; # 2^32 + 100,000
+command_ok([ 'pg_resetwal', '--next-oid' => $new_nextoid, $node->data_dir ],
+ 'runs with --next-oid value above 2^32');
+command_like(
+ [ 'pg_resetwal', '--dry-run', $node->data_dir ],
+ qr/^Latest checkpoint's NextOID: *$new_nextoid$/m,
+ '8-byte --next-oid value preserved in pg_control');
+$node->start;
+is( $node->safe_psql(
+ 'postgres',
+ "SELECT next_oid >= '$new_nextoid'::oid8 FROM pg_control_checkpoint()"
+ ),
+ 't',
+ 'new 8-byte OID reported');
+
done_testing();
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 7543a9880451..f0fa2b1689fa 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -245,7 +245,7 @@ get_control_data(ClusterInfo *cluster)
pg_fatal("%d: controldata retrieval problem", __LINE__);
p++; /* remove ':' char */
- cluster->controldata.chkpnt_nxtoid = str2uint(p);
+ cluster->controldata.chkpnt_nxtoid = str2uint64(p);
got_oid = true;
}
else if ((p = strstr(bufin, "Latest checkpoint's NextMultiXactId:")) != NULL)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 7366fd4627cc..3704c620603c 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -196,7 +196,7 @@ main(int argc, char **argv)
*/
prep_status("Setting next OID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
- "\"%s/pg_resetwal\" -o %u \"%s\"",
+ "\"%s/pg_resetwal\" -o " OID8_FORMAT " \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtoid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index c80e8fb4031c..4cb79bb780e6 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -216,7 +216,7 @@ typedef struct
char nextxlogfile[25];
uint32 chkpnt_nxtxid;
uint32 chkpnt_nxtepoch;
- uint32 chkpnt_nxtoid;
+ Oid8 chkpnt_nxtoid;
uint32 chkpnt_nxtmulti;
uint64 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index a5cfd53265d4..728fd6b0b120 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -588,6 +588,20 @@ $oldnode->start;
$oldnode->safe_psql('postgres', 'DROP DATABASE regression_invalid');
$oldnode->stop;
+# The OID counter is 8 bytes wide, check that it is carried. Older versions
+# may not support 8-byte OIDs, so skip in this case.
+my $big_next_oid = '4295067296'; # 2^32 + 100000
+if (!defined($ENV{oldinstall}))
+{
+ command_ok(
+ [
+ 'pg_resetwal',
+ '--next-oid' => $big_next_oid,
+ $oldnode->data_dir
+ ],
+ 'set an 8-byte OID counter in the old instance');
+}
+
# --check command works here, cleans up pg_upgrade_output.d.
command_ok(
[
@@ -624,6 +638,17 @@ ok( !-d $newnode->data_dir . "/pg_upgrade_output.d",
$newnode->start;
+# The 8-byte OID has been carried.
+if (!defined($ENV{oldinstall}))
+{
+ is( $newnode->safe_psql(
+ 'postgres',
+ "SELECT next_oid >= '$big_next_oid'::oid8 FROM pg_control_checkpoint()"
+ ),
+ 't',
+ '8-byte OID counter is carried over by pg_upgrade');
+}
+
# Check if there are any logs coming from pg_upgrade, that would only be
# retained on failure.
my $log_path = $newnode->data_dir . "/pg_upgrade_output.d";
diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml
index 1ff023b741c8..e56c9a22c42a 100644
--- a/doc/src/sgml/func/func-info.sgml
+++ b/doc/src/sgml/func/func-info.sgml
@@ -3433,7 +3433,7 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
<row>
<entry><structfield>next_oid</structfield></entry>
- <entry><type>oid</type></entry>
+ <entry><type>oid8</type></entry>
</row>
<row>
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index 664c525f481c..a9580f35de95 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -286,11 +286,11 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>-o <replaceable class="parameter">oid</replaceable></option></term>
- <term><option>--next-oid=<replaceable class="parameter">oid</replaceable></option></term>
+ <term><option>-o <replaceable class="parameter">oid8</replaceable></option></term>
+ <term><option>--next-oid=<replaceable class="parameter">oid8</replaceable></option></term>
<listitem>
<para>
- Manually set the next OID.
+ Manually set the next OID (8 bytes).
</para>
<para>
--
2.55.0
v15-0002-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch (text/plain, 16.1K) download
From 2a5e6c3fc4f27e5bfa417bf189e1154646af56a0 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 13:40:41 +0900
Subject: [PATCH v15 02/11] Refactor some TOAST value ID code to use Oid8
instead of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 23 +++++---
contrib/amcheck/verify_heapam.c | 55 +++++++++++--------
6 files changed, 60 insertions(+), 44 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index b2346fd75bd4..aaae6e0fef69 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index ff03a2b816fc..ebec31a22e40 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1987,7 +1987,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 0f234661bfa9..ca5ed8dfabca 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);
-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)
+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)
+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 3b7c658a90e6..81154c17376c 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -729,7 +729,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",
+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -741,13 +741,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -756,7 +756,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",
+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -785,7 +785,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",
+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 457d544eec93..dd4b6e8f7f45 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */
+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -5010,7 +5010,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5034,8 +5034,10 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;
+ Oid8 chunk_id;
int32 chunk_seq;
+ Oid valueid_type;
+ Datum valueid_datum;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
@@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));
+ valueid_type = TupleDescAttr(desc, 0)->atttypid;
+ valueid_datum = fastgetattr(newtup, 1, desc, &isnull);
+ if (valueid_type == OID8OID)
+ chunk_id = DatumGetObjectId8(valueid_datum);
+ else
+ chunk_id = DatumGetObjectId(valueid_datum);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
@@ -5061,11 +5068,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5174,6 +5181,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5194,13 +5202,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,
+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index bbaaec89f8ae..33b00fa4dc5c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1564,9 +1564,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 chunksize;
int32 expected_size;
int32 max_chunk_size;
+ Oid8 toast_valueid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1574,16 +1576,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",
+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",
+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1594,8 +1596,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",
+ toast_valueid,
chunk_seq));
return;
}
@@ -1614,8 +1616,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",
+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1626,8 +1628,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",
+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1637,8 +1639,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",
+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1669,6 +1671,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external_oid toast_pointer;
@@ -1777,12 +1780,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",
- toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1809,16 +1813,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",
- toast_pointer.va_valueid, cmid));
+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",
+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",
+ toast_pointer_valueid));
return true;
}
@@ -1826,8 +1830,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",
+ toast_pointer_valueid));
return true;
}
@@ -1873,6 +1877,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1903,14 +1908,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;
+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " not found in toast table",
+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",
+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.55.0
v15-0003-Switch-pg_column_toast_chunk_id-return-value-fro.patch (text/plain, 5.7K) download
From 79c4d680b830803acbd02cc5672dba0997fe2ebf Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v15 03/11] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index c53ce68c7173..267eaee29806 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7907,7 +7907,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',
+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index e5632d930682..1e264904a6e1 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);
+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out
index cb14ddcee34b..21859fc0dce2 100644
--- a/src/test/isolation/expected/cluster-toast-value-reuse.out
+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out
@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec
index 9a2d10600b39..96cf899db296 100644
--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec
+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec
@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
index fa76bcf28ef9..cc8f034d0167 100644
--- a/src/test/modules/injection_points/specs/repack_toast.spec
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);
+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2990e0c4f286..35d6ee2ce9eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 950d9ab1a4a1..94b91964c046 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 54eeb42e5bce..70d762e73328 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>
+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.55.0
v15-0004-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch (text/plain, 9.3K) download
From d0873e44ebadd89ed33d76e0d8817d851ceda046 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 10:57:59 +0900
Subject: [PATCH v15 04/11] Add support for TOAST chunk_id type in binary
upgrades
This commit adds a new function, which would set the type of a chunk_id
attribute for a TOAST table across upgrades. This piece currently works
only with chunk_id = OIDOID, but it is required in a follow-up patch
where support for chunk_id = OID8OID is supported on top of the existing
one.
---
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 ++++
src/backend/catalog/heap.c | 1 +
src/backend/catalog/toasting.c | 20 ++++++++++++++++++-
src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++++
src/bin/pg_dump/pg_dump.c | 10 +++++++++-
.../expected/spgist_name_ops.out | 6 ++++--
7 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae443859..117f3e3f7746 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -29,6 +29,7 @@ extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_index_pg_class_relfilenumber;
extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber;
+extern PGDLLIMPORT Oid binary_upgrade_next_toast_chunk_id_typoid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 267eaee29806..02602d96a2be 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11992,6 +11992,10 @@
proname => 'binary_upgrade_set_next_toast_pg_class_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_toast_pg_class_oid' },
+{ oid => '8219', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_toast_chunk_id_typoid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_toast_chunk_id_typoid' },
{ oid => '3589', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_set_next_pg_enum_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index a107193d7ab3..1c188b7a0ff3 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -80,6 +80,7 @@
/* Potentially set by pg_upgrade_support functions */
Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid;
Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid;
+Oid binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
RelFileNumber binary_upgrade_next_heap_pg_class_relfilenumber = InvalidRelFileNumber;
RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber = InvalidRelFileNumber;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 4aa52a4bd253..bc0a72268335 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -146,6 +146,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_chunkid_typid = OIDOID;
/*
* Is it already toasted?
@@ -184,6 +185,23 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!OidIsValid(binary_upgrade_next_toast_pg_class_oid))
return false;
+
+ /*
+ * The attribute type for chunk_id should have been set when requesting
+ * a TOAST table creation.
+ */
+ if (!OidIsValid(binary_upgrade_next_toast_chunk_id_typoid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("toast chunk_id type not set while in binary upgrade mode")));
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
+ binary_upgrade_next_toast_chunk_id_typoid)));
+
+ toast_chunkid_typid = binary_upgrade_next_toast_chunk_id_typoid;
+ binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
}
/*
@@ -205,7 +223,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,
+ toast_chunkid_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4feeb..06b789d71ded 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -149,6 +149,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_toast_chunk_id_typoid(PG_FUNCTION_ARGS)
+{
+ Oid typoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_toast_chunk_id_typoid = typoid;
+
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7a12e03ca028..bb17a2b52400 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -104,6 +104,7 @@ typedef struct
RelFileNumber relfilenumber; /* object filenode */
Oid toast_oid; /* toast table OID */
RelFileNumber toast_relfilenumber; /* toast table filenode */
+ Oid toast_chunk_id_typoid; /* type of chunk_id attribute */
Oid toast_index_oid; /* toast table index OID */
RelFileNumber toast_index_relfilenumber; /* toast table index filenode */
} BinaryUpgradeClassOidItem;
@@ -5869,7 +5870,10 @@ collectBinaryUpgradeClassOids(Archive *fout)
const char *query;
query = "SELECT c.oid, c.relkind, c.relfilenode, c.reltoastrelid, "
- "ct.relfilenode, i.indexrelid, cti.relfilenode "
+ "ct.relfilenode, i.indexrelid, cti.relfilenode, "
+ "(SELECT a.atttypid FROM pg_attribute AS a "
+ " WHERE a.attrelid = c.reltoastrelid AND attname = 'chunk_id'::text) "
+ " AS toastchunktypid "
"FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_index i "
"ON (c.reltoastrelid = i.indrelid AND i.indisvalid) "
"LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) "
@@ -5891,6 +5895,7 @@ collectBinaryUpgradeClassOids(Archive *fout)
binaryUpgradeClassOids[i].toast_relfilenumber = atooid(PQgetvalue(res, i, 4));
binaryUpgradeClassOids[i].toast_index_oid = atooid(PQgetvalue(res, i, 5));
binaryUpgradeClassOids[i].toast_index_relfilenumber = atooid(PQgetvalue(res, i, 6));
+ binaryUpgradeClassOids[i].toast_chunk_id_typoid = atooid(PQgetvalue(res, i, 7));
}
PQclear(res);
@@ -5955,6 +5960,9 @@ binary_upgrade_set_pg_class_oids(Archive *fout,
appendPQExpBuffer(upgrade_buffer,
"SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n",
entry->toast_relfilenumber);
+ appendPQExpBuffer(upgrade_buffer,
+ "SELECT pg_catalog.binary_upgrade_set_next_toast_chunk_id_typoid('%u'::pg_catalog.oid);\n",
+ entry->toast_chunk_id_typoid);
/* every toast table has an index */
appendPQExpBuffer(upgrade_buffer,
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede2430..35e59d0cd83c 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -61,9 +61,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -110,9 +111,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.55.0
v15-0005-Add-relation-option-toast_value_type.patch (text/plain, 6.4K) download
From 296ca1549a4aff6ecc4c2bac4602335f1a3566b2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:54:58 +0900
Subject: [PATCH v15 05/11] Add relation option toast_value_type
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is created
initially. This parameter has no effect if a TOAST table exists, even
after it is modified later on, even on rewrites.
This can be set only to "oid" currently, and will be expanded with a
second mode later.
Note: perhaps it would make sense to introduce that only when support
for 8-byte OID values are added to TOAST, the split is here to ease
review.
---
src/include/utils/rel.h | 17 +++++++++++++++++
src/backend/access/common/reloptions.c | 21 +++++++++++++++++++++
src/backend/catalog/toasting.c | 6 ++++++
src/bin/psql/tab-complete.in.c | 1 +
doc/src/sgml/ref/create_table.sgml | 18 ++++++++++++++++++
5 files changed, 63 insertions(+)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 41ab4586c6b5..f9e26ce6654a 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -341,11 +341,20 @@ typedef enum StdRdOptIndexCleanup
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
} StdRdOptIndexCleanup;
+/* StdRdOptions->toast_value_type values */
+typedef enum StdRdOptToastValueType
+{
+ STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+} StdRdOptToastValueType;
+
typedef struct StdRdOptions
{
int32 vl_len_; /* varlena header (do not touch directly!) */
int fillfactor; /* page fill factor in percent (0..100) */
int toast_tuple_target; /* target for tuple toasting */
+ StdRdOptToastValueType toast_value_type; /* type assigned to chunk_id
+ * at toast table creation */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
int parallel_workers; /* max number of parallel workers */
@@ -370,6 +379,14 @@ typedef struct StdRdOptions
((relation)->rd_options ? \
((StdRdOptions *) (relation)->rd_options)->toast_tuple_target : (defaulttarg))
+/*
+ * RelationGetToastValueType
+ * Returns the relation's toast_value_type. Note multiple eval of argument!
+ */
+#define RelationGetToastValueType(relation, defaulttarg) \
+ ((relation)->rd_options ? \
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)
+
/*
* RelationGetFillFactor
* Returns the relation's fillfactor. Note multiple eval of argument!
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index c3f6f45f4609..82491a31a9a4 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -548,6 +548,14 @@ static relopt_enum_elt_def viewCheckOptValues[] =
{(const char *) NULL} /* list terminator */
};
+/* values from StdRdOptToastValueType */
+static relopt_enum_elt_def StdRdOptToastValueTypes[] =
+{
+ /* no value for INVALID */
+ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {(const char *) NULL} /* list terminator */
+};
+
static relopt_enum enumRelOpts[] =
{
{
@@ -561,6 +569,17 @@ static relopt_enum enumRelOpts[] =
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")
},
+ {
+ {
+ "toast_value_type",
+ "Controls the attribute type of chunk_id at toast table creation",
+ RELOPT_KIND_HEAP,
+ ShareUpdateExclusiveLock
+ },
+ StdRdOptToastValueTypes,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ gettext_noop("Valid values are \"oid\".")
+ },
{
{
"buffering",
@@ -2078,6 +2097,8 @@ static const relopt_parse_elt stdRdOptionsTab[] = {
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_analyze_min_duration)},
{"toast_tuple_target", RELOPT_TYPE_INT,
offsetof(StdRdOptions, toast_tuple_target)},
+ {"toast_value_type", RELOPT_TYPE_ENUM,
+ offsetof(StdRdOptions, toast_value_type)},
{"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL,
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)},
{"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL,
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index bc0a72268335..28ddbd10d357 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -159,9 +159,15 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!IsBinaryUpgrade)
{
+ StdRdOptToastValueType value_type;
+
/* Normal mode, normal check */
if (!needs_toast_table(rel))
return false;
+
+ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
+ if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
+ toast_chunkid_typid = OIDOID;
}
else
{
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 8e1a1d69740c..84a129efff54 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -1441,6 +1441,7 @@ static const char *const table_storage_parameters[] = {
"toast.vacuum_max_eager_freeze_failure_rate",
"toast.vacuum_truncate",
"toast_tuple_target",
+ "toast_value_type",
"user_catalog_table",
"vacuum_index_cleanup",
"vacuum_max_eager_freeze_failure_rate",
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index e342585c7f08..3da754045c04 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1653,6 +1653,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry id="reloption-toast-value-type" xreflabel="toast_value_type">
+ <term><literal>toast_value_type</literal> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>toast_value_type</varname> storage parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ The toast_value_type specifies the attribute type of
+ <literal>chunk_id</literal> used when initially creating a toast
+ relation for this table.
+ By default this parameter is <literal>oid</literal>, to assign
+ <type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter cannot be set for TOAST tables.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="reloption-parallel-workers" xreflabel="parallel_workers">
<term><literal>parallel_workers</literal> (<type>integer</type>)
<indexterm>
--
2.55.0
v15-0006-Add-support-for-oid8-TOAST-values.patch (text/plain, 17.4K) download
From a8f000c92bae94d41559f933c27c56f134b30356 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:45:13 +0900
Subject: [PATCH v15 06/11] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, casted to OID (for now).
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 3 +-
src/backend/access/common/toast_internals.c | 94 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 19 ++++-
src/backend/catalog/toasting.c | 24 +++++-
doc/src/sgml/ref/create_table.sgml | 2 +
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 22 +++--
9 files changed, 133 insertions(+), 42 deletions(-)
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f9e26ce6654a..b7da03446a94 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 82491a31a9a4..ea9a04179090 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
@@ -578,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")
+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..2c180491c2ff 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,8 +132,10 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -200,24 +203,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OIDOID)
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ Assert(false);
}
else
{
@@ -263,17 +274,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OIDOID)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
}
}
@@ -303,7 +319,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OIDOID)
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ else if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -384,6 +403,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -395,6 +415,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -405,10 +427,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +485,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +493,24 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..a93e98788da7 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +658,18 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 28ddbd10d357..66d483bb1be3 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -168,6 +169,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
toast_chunkid_typid = OIDOID;
+ else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)
+ toast_chunkid_typid = OID8OID;
}
else
{
@@ -200,7 +203,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -225,6 +229,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -343,7 +360,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OIDOID)
+ opclassIds[0] = OID_BTREE_OPS_OID;
+ else if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3da754045c04..3ed8f3581177 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1666,6 +1666,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
relation for this table.
By default this parameter is <literal>oid</literal>, to assign
<type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter can be set to <type>oid8</type> to use <type>oid8</type>
+ as attribute type for <literal>chunk_id</literal>.
This parameter cannot be set for TOAST tables.
</para>
</listitem>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 90aae3defcbc..21bd67c11ea0 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an OID8 identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 33b00fa4dc5c..42f409548e1d 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1876,8 +1876,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1885,10 +1889,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v15-0007-Add-tests-for-TOAST-relations-with-oid8-as-value.patch (text/plain, 21.1K) download
From a963f46f2071206a20f3b0b229ec9bd4b8d6de4c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 08:22:56 +0900
Subject: [PATCH v15 07/11] Add tests for TOAST relations with oid8 as value
type
This adds coverage for relations created with default_toast_type =
'oid8', for external TOAST pointers both compressed and uncompressed.
---
src/test/regress/expected/strings.out | 257 ++++++++++++++++++++++----
src/test/regress/sql/strings.sql | 156 ++++++++++++----
2 files changed, 338 insertions(+), 75 deletions(-)
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 313c50394650..080cb039ce43 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2105,7 +2198,97 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 38946e8954df..3e7459d29ff1 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,169 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
-- test internally compressing datums
--
2.55.0
v15-0008-More-test-coverage-for-TOAST-relations-with-oid8.patch (text/plain, 13.4K) download
From 043fcf768f069908bce9a26ac7b398edcf7eaa2a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:46:39 +0900
Subject: [PATCH v15 08/11] More test coverage for TOAST relations with oid8 as
chunk_id
This covers more area of the code in terms of tests, where oid8 vartags
are added:
- amcheck
- test_decoding
- reloptions, with various incorrect patterns for toast_value_type.
---
src/test/regress/expected/reloptions.out | 35 +++++++++++++++++++++
src/test/regress/sql/reloptions.sql | 20 ++++++++++++
contrib/amcheck/expected/check_heap.out | 40 ++++++++++++++++++++++++
contrib/amcheck/sql/check_heap.sql | 27 ++++++++++++++++
contrib/test_decoding/expected/toast.out | 39 +++++++++++++++++++++++
contrib/test_decoding/sql/toast.sql | 20 ++++++++++++
6 files changed, 181 insertions(+)
diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out
index e3a974f26112..6a50713fb58e 100644
--- a/src/test/regress/expected/reloptions.out
+++ b/src/test/regress/expected/reloptions.out
@@ -194,6 +194,41 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid;
-- Fail on non-existent options in toast namespace
CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42);
ERROR: unrecognized parameter "not_existing_option"
+-- Test toast_value_type.
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid8');
+ERROR: unrecognized parameter "toast_value_type"
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
+ERROR: invalid value for enum option "toast_value_type": int8
+DETAIL: Valid values are "oid" and "oid8".
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid8');
+SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+ reloptions
+-------------------------
+ {toast_value_type=oid8}
+(1 row)
+
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ chunk_id_type
+---------------
+ oid8
+(1 row)
+
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
+DROP TABLE reloptions_test2;
+-- Non-supported relkinds.
+CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
+ WITH (toast_value_type = 'oid8');
+ERROR: unrecognized parameter "toast_value_type"
+CREATE TABLE reloptions_test2 (i int) PARTITION BY RANGE (i)
+ WITH (toast_value_type = 'oid8');
+ERROR: cannot specify storage parameters for a partitioned table
+HINT: Specify storage parameters for its leaf partitions instead.
-- Mix TOAST & heap
DROP TABLE reloptions_test;
CREATE TABLE reloptions_test (s VARCHAR) WITH
diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql
index 680c8bf86148..53cad02f2a28 100644
--- a/src/test/regress/sql/reloptions.sql
+++ b/src/test/regress/sql/reloptions.sql
@@ -112,6 +112,26 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid;
-- Fail on non-existent options in toast namespace
CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42);
+-- Test toast_value_type.
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid8');
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid8');
+SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
+DROP TABLE reloptions_test2;
+-- Non-supported relkinds.
+CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
+ WITH (toast_value_type = 'oid8');
+CREATE TABLE reloptions_test2 (i int) PARTITION BY RANGE (i)
+ WITH (toast_value_type = 'oid8');
+
-- Mix TOAST & heap
DROP TABLE reloptions_test;
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index 979e5e84e723..8686fba712d0 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -199,6 +199,44 @@ SELECT * FROM verify_heapam('test_partition',
-------+--------+--------+-----
(0 rows)
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+ relname | chunk_id_type
+-----------------+---------------
+ test_toast_oid | oid
+ test_toast_oid8 | oid8
+(2 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -232,6 +270,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 1745bae634e5..eb46fe710351 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -112,6 +112,31 @@ SELECT * FROM verify_heapam('test_partition',
startblock := NULL,
endblock := NULL);
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -139,6 +164,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
endblock := NULL);
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.out
index a757e7dc8d54..8bef6cc32af4 100644
--- a/contrib/test_decoding/expected/toast.out
+++ b/contrib/test_decoding/expected/toast.out
@@ -382,6 +382,45 @@ SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot',
COMMIT
(4 rows)
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ shortened | len
+------------------------------------------------------------------------------------+--------
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:1 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:2 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: old-key: id[integer]:1 ne..nchanged-toast-datum | 116
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: id[integer]:2 data[text]:..bcdefghijabcdefghij' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: DELETE: id[integer]:11 | 49
+ table public.toasted_oid8: DELETE: id[integer]:2 | 48
+ COMMIT | 6
+(16 rows)
+
+DROP TABLE toasted_oid8;
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sql
index d1c560a174d6..8a2c49a5c9a7 100644
--- a/contrib/test_decoding/sql/toast.sql
+++ b/contrib/test_decoding/sql/toast.sql
@@ -324,4 +324,24 @@ INSERT INTO tbl2 VALUES(1);
commit;
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+DROP TABLE toasted_oid8;
+
SELECT pg_drop_replication_slot('regression_slot');
--
2.55.0
v15-0009-More-test-coverage-for-oid8-with-compression-and.patch (text/plain, 8.4K) download
From 0426e7499bd2b567064a0d657002d83e77e38e6c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 08:43:35 +0900
Subject: [PATCH v15 09/11] More test coverage for oid8 with compression and
out-of-line values
These two gaps have been pointed out by Claude, for the following items:
- pg_column_compression() with oid8 vartags
- More interesting: UPDATE with out-of-line TOAST datum that belongs to
a different TOAST table, where toast_tuple_init() may try to compare two
TOAST tuples with a different vartag once the oid8 flavor is introduced.
As of the current state of the code, all vartags are OID-based. A
follow-up patch will complicate that.
---
src/test/regress/expected/compression.out | 41 +++++++++++++++++++++
src/test/regress/expected/strings.out | 43 +++++++++++++++++++++++
src/test/regress/sql/compression.sql | 22 ++++++++++++
src/test/regress/sql/strings.sql | 25 +++++++++++++
4 files changed, 131 insertions(+)
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 09f198149aa4..e9128193515a 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
(1 row)
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+ uncompressed
+--------------
+ t
+(1 row)
+
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+ uncompressed
+--------------
+ t
+(1 row)
+
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+DROP TABLE toastcomp_oid, toastcomp_oid8;
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 080cb039ce43..7e3a641259c2 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -2289,6 +2289,49 @@ ALTER TABLE toasttest_oid8 RESET (toast_value_type);
-- Reset column storage to its default
ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index ce5ea37a660c..229143f06ddd 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2;
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+DROP TABLE toastcomp_oid, toastcomp_oid8;
+
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 3e7459d29ff1..4d6a1e02359c 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -727,6 +727,31 @@ ALTER TABLE toasttest_oid8 RESET (toast_value_type);
ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+
+DROP TABLE toastupd_oid, toastupd_oid8;
+
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
--
2.55.0
v15-0010-Add-support-for-TOAST-pointers-as-oid8.patch (text/plain, 51.0K) download
From a39c74131923f0e62e8339eff31f46adf439aa61 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:48:29 +0900
Subject: [PATCH v15 10/11] Add support for TOAST pointers as oid8
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 3 +
src/include/access/heaptoast.h | 16 +-
src/include/utils/rel.h | 2 +-
src/include/varatt.h | 93 ++++-
src/backend/access/common/detoast.c | 151 ++++++--
src/backend/access/common/toast_compression.c | 11 +-
src/backend/access/common/toast_internals.c | 353 ++++++++++++------
src/backend/access/heap/heaptoast.c | 17 +-
src/backend/access/table/toast_helper.c | 4 +
src/backend/catalog/toasting.c | 19 +-
.../replication/logical/reorderbuffer.c | 45 ++-
src/backend/utils/adt/varlena.c | 19 +-
doc/src/sgml/ref/create_table.sgml | 25 +-
doc/src/sgml/storage.sgml | 10 +-
contrib/amcheck/verify_heapam.c | 90 +++--
15 files changed, 631 insertions(+), 227 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..f065ea034e98 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -30,6 +30,9 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..b7ef69a8f56b 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,13 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b7da03446a94..8bec73d55794 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -386,7 +386,7 @@ typedef struct StdRdOptions
*/
#define RelationGetToastValueType(relation, defaulttarg) \
((relation)->rd_options ? \
- ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : (defaulttarg))
/*
* RelationGetFillFactor
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 33979bbaaad1..1349a54371db 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -38,6 +38,44 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except for the value ID, which is 8 bytes wide. The value ID is split
+ * into two uint32 to force alignment.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==
+ sizeof(varatt_external_oid8),
+ "varatt_external_oid8 should have no padding");
+
+/* Get/set the 64-bit value ID from the lo/hi halves */
+static inline Oid8
+VARATT_EXTERNAL_OID8_GET_VALUEID(varatt_external_oid8 toast_pointer)
+{
+ return ((Oid8) toast_pointer.va_valueid_lo) |
+ (((Oid8) toast_pointer.va_valueid_hi) << 32);
+}
+
+static inline void
+VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id)
+{
+ toast_pointer->va_valueid_lo = (uint32) id;
+ toast_pointer->va_valueid_hi = (uint32) (id >> 32);
+}
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -87,6 +125,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -108,6 +147,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -360,10 +401,29 @@ VARATT_IS_EXTERNAL(const void *PTR)
/* Is varlena datum a pointer to on-disk toasted data? */
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
+{
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
+}
+
+/* Is varlena datum a pointer to on-disk toasted data with 4-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID(const void *PTR)
{
return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
+{
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
+}
+
/* Is varlena datum an indirect pointer? */
static inline bool
VARATT_IS_EXTERNAL_INDIRECT(const void *PTR)
@@ -519,16 +579,6 @@ VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
* Testing whether an externally-stored value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -543,4 +593,27 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid8.
+ */
+static inline Size
+VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+}
+
+static inline uint32
+VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+}
+
+static inline bool
+VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)
+{
+ return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <
+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+}
+
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..393135f1b9f3 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,12 +225,32 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /* Extract extsize and compression info from the appropriate pointer */
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +260,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,14 +271,9 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,23 +359,47 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- result = (varlena *) palloc(attrsize + VARHDRSZ);
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
+
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -369,10 +408,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +437,43 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +486,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +499,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +508,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +609,20 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +679,20 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 3b1bd6519261..200cee6be705 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -260,7 +260,16 @@ toast_get_compression_id(varlena *attr)
* the external toast pointer. If compressed inline, fetch it from the
* toast compression header.
*/
- if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ cmid = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ }
+ else if (VARATT_IS_EXTERNAL_ONDISK_OID(attr))
{
varatt_external_oid toast_pointer;
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 2c180491c2ff..84c4361af23d 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -125,7 +125,6 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -133,9 +132,17 @@ toast_save_datum(Relation rel, Datum value,
int num_indexes;
int validIndex;
Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
Assert(OidIsValid(toast_typid));
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -165,28 +172,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -198,13 +209,14 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or OID8 TOAST relations.
+ * Choose a new value to use as the value ID for this toast value, and
+ * determine the maximum chunk size based on the TOAST table's chunk_id
+ * type.
*
* Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
@@ -213,84 +225,150 @@ toast_save_datum(Relation rel, Datum value,
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * conflict with either new or existing toast value IDs. For Oid8 tables,
+ * value conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OIDOID)
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
+ if (!OidIsValid(rel->rd_toastoid))
+ {
+ /* normal case: just choose a new Oid8 */
+ va_valueid = GetNewObjectId8();
+ }
else
- Assert(false);
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid8;
+ if (oldexternal != NULL)
+ {
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+
+ /*
+ * Only consider the old pointer if its vartag matches the
+ * kind of pointer we are about to build.
+ */
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(oldexternal))
+ {
+ varatt_external_oid8 old_toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /*
+ * This value came from the old toast table; reuse its
+ * ID.
+ */
+ va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);
+
+ /*
+ * There is a corner case here: the table rewrite
+ * might have to copy both live and recently-dead
+ * versions of a row, and those versions could easily
+ * reference the same toast value. When we copy the
+ * second or later version of such a row, reusing the
+ * OID will mean we select an OID that's already in the
+ * new toast table. Check for that, and if so, just
+ * fall through without writing the data again.
+ *
+ * While annoying and ugly-looking, this is a good
+ * thing because it ensures that we wind up with only
+ * one copy of the toast value when there is only one
+ * copy in the old toast table. Before we detected
+ * this case, we'd have made multiple copies, wasting
+ * space; and what's worse, the copies belonging to
+ * already-deleted heap tuples would not be reclaimed
+ * by VACUUM.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /*
+ * Match, so short-circuit the data storage loop
+ * below.
+ */
+ data_todo = 0;
+ }
+ }
+ }
+ }
+ if (va_valueid == InvalidOid8)
+ {
+ /*
+ * New value. For Oid8 we just pick a new ID without worrying
+ * about conflicts.
+ */
+ va_valueid = GetNewObjectId8();
+ }
+ }
+
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
+ if (!OidIsValid(rel->rd_toastoid))
{
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ /* normal case: just choose an unused OID */
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
+ else
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid;
+ if (oldexternal != NULL)
{
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
+ /* Only reuse a pointer of the matching type; same as above */
+ if (VARATT_IS_EXTERNAL_ONDISK_OID(oldexternal))
{
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
+ varatt_external_oid old_toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /*
+ * This value came from the old toast table; reuse its
+ * OID.
+ */
+ va_valueid = old_toast_pointer.va_valueid;
+
+ /*
+ * Corner case during table rewrite with multiple
+ * versions of the same row. See above for details.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /*
+ * Match, so short-circuit the data storage loop
+ * below.
+ */
+ data_todo = 0;
+ }
+ }
}
}
- }
- if (toast_pointer.va_valueid == InvalidOid)
- {
- /*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
- */
- if (toast_typid == OIDOID)
+ if (va_valueid == InvalidOid)
{
+ /*
+ * new value; must choose a value that doesn't conflict in
+ * either old or new toast table.
+ */
do
{
- toast_pointer.va_valueid =
+ va_valueid =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ va_valueid));
}
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
}
+
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
/*
@@ -314,15 +392,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
- if (toast_typid == OIDOID)
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
- else if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -376,11 +454,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(&toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -395,7 +496,6 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -403,42 +503,75 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toast_typid;
+ Oid toastrelid;
+ vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
/*
- * Open the toast relation and its indexes
+ * Determine the pointer type from the datum's vartag and extract the
+ * toast relation OID and value ID accordingly. The vartag tells us
+ * everything we need - no TOAST table schema lookup required.
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
- toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ tag = VARTAG_EXTERNAL(attr);
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
+ if (tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- if (toast_typid == OIDOID)
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toastrelid = toast_pointer8.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ Assert(tag == VARTAG_ONDISK_OID);
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toastrelid = toast_pointer.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(toast_pointer.va_valueid));
- else
- Assert(false);
+ }
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -499,18 +632,16 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
- else if (toast_typid == OID8OID)
+ if (toast_typid == OID8OID)
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
ObjectId8GetDatum(valueid));
else
- Assert(false);
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index a93e98788da7..31e6115de201 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -650,7 +650,10 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ if (toast_typid == OID8OID)
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+ else
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
@@ -658,18 +661,16 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
- else if (toast_typid == OID8OID)
+ if (toast_typid == OID8OID)
ScanKeyInit(&toastkey[0],
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
ObjectId8GetDatum(valueid));
else
- Assert(false);
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6f2d691cc681..6c7257f5f94b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc)
/*
* If the old value is stored on disk, check if it has changed so
* we have to delete it later.
+ *
+ * Note that TOAST pointers could have different vartags, for oid
+ * or oid8, and these can have a different sizes.
*/
if (att->attlen == -1 && !ttc->ttc_oldisnull[i] &&
VARATT_IS_EXTERNAL_ONDISK(old_value))
{
if (ttc->ttc_isnull[i] ||
!VARATT_IS_EXTERNAL_ONDISK(new_value) ||
+ VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) ||
memcmp(old_value, new_value,
VARSIZE_EXTERNAL(old_value)) != 0)
{
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 66d483bb1be3..57ebaec69c78 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -167,10 +167,21 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
return false;
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
- if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
- toast_chunkid_typid = OIDOID;
- else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)
- toast_chunkid_typid = OID8OID;
+
+ /* no default clause to catch new values added */
+ switch (value_type)
+ {
+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID:
+ toast_chunkid_typid = OIDOID;
+ break;
+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID8:
+ toast_chunkid_typid = OID8OID;
+ break;
+ case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID:
+ elog(ERROR, "unexpected toast_value_type value %d",
+ value_type);
+ break;
+ }
}
else
{
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dd4b6e8f7f45..086ef31e4f2f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5176,12 +5176,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* va_rawsize is the size of the original datum -- including header */
varatt_external_oid toast_pointer;
+ varatt_external_oid8 toast_pointer8;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
+ int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5198,11 +5200,21 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena_pointer = (varlena *) DatumGetPointer(attrs[natt]);
/* no need to do anything if the tuple isn't external */
- if (!VARATT_IS_EXTERNAL(varlena_pointer))
+ if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);
+ toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ rawsize = toast_pointer8.va_rawsize;
+ }
+ else
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
+ rawsize = toast_pointer.va_rawsize;
+ }
/*
* Check whether the toast tuple changed, replace if so.
@@ -5220,7 +5232,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(rawsize);
ent->reconstructed = reconstructed;
@@ -5245,13 +5257,28 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ /* Verify size and set compression status based on pointer type */
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 1e264904a6e1..e104a6308d65 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,7 +4261,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external_oid toast_pointer;
+ Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4288,9 +4288,22 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ {
+ varatt_external_oid8 toast_pointer;
- PG_RETURN_OID8(toast_pointer.va_valueid);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_valueid;
+ }
+
+ PG_RETURN_OID8(result);
}
/*
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3ed8f3581177..158f79e7135f 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1661,14 +1661,23 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- The toast_value_type specifies the attribute type of
- <literal>chunk_id</literal> used when initially creating a toast
- relation for this table.
- By default this parameter is <literal>oid</literal>, to assign
- <type>oid</type> as attribute type to <literal>chunk_id</literal>.
- This parameter can be set to <type>oid8</type> to use <type>oid8</type>
- as attribute type for <literal>chunk_id</literal>.
- This parameter cannot be set for TOAST tables.
+ Specifies the attribute type of <literal>chunk_id</literal> to use when
+ creating a <acronym>TOAST</acronym> relation for this table. The
+ default is <literal>oid</literal>; <literal>oid8</literal> assigns
+ <type>oid8</type> instead, which allows a larger number of distinct
+ <acronym>TOAST</acronym>ed values at the price of four extra bytes per
+ out-of-line pointer. This parameter cannot be set for
+ <acronym>TOAST</acronym> tables.
+ </para>
+ <para>
+ The value is only consulted when the <acronym>TOAST</acronym> relation
+ is created. Changing it afterwards has no effect on an existing
+ <acronym>TOAST</acronym> relation, and a table rewrite such as
+ <link linkend="sql-cluster"><command>CLUSTER</command></link> or
+ <link linkend="sql-vacuum"><command>VACUUM FULL</command></link>
+ preserves the type already in use. A dump and restore, on the other
+ hand, recreates the <acronym>TOAST</acronym> relation and therefore
+ applies the current value of this parameter.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 21bd67c11ea0..3561a6a67ece 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,9 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the
+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 42f409548e1d..86dc9cf51a93 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1566,9 +1568,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 max_chunk_size;
Oid8 toast_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ toast_valueid = ta->va_valueid;
+
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1672,8 +1677,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1732,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1777,26 +1784,46 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
/*
- * Must copy attr into toast_pointer for alignment considerations
+ * Must copy attr into toast_pointer for alignment considerations.
+ * Branch on the tag to determine which pointer type to extract.
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ va_tag_value = VARTAG_EXTERNAL(attr);
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ va_rawsize = toast_pointer8.va_rawsize;
+ va_extinfo = toast_pointer8.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
+ va_rawsize = toast_pointer.va_rawsize;
+ va_extinfo = toast_pointer.va_extinfo;
+ }
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1850,7 +1877,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ /* The pointer has already been decoded above, just reuse it */
+ ta->tag = va_tag_value;
+ ta->va_valueid = toast_pointer_valueid;
+ ta->va_extinfo = va_extinfo;
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1878,29 +1908,33 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
int32 max_chunk_size;
Oid8 toast_valueid;
- Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
+ if (ta->tag == VARTAG_ONDISK_OID8)
+ {
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ ObjectId8GetDatum(toast_valueid));
+ }
else
- Assert(false);
+ {
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) toast_valueid));
+ }
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1920,8 +1954,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
--
2.55.0
v15-0011-Fix-toast_tuple_find_biggest_attribute-for-OID8-.patch (text/plain, 6.7K) download
From 9a2cdce244f142734dbc925abbe769abdecd8a28 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 15:32:34 +0900
Subject: [PATCH v15 11/11] Fix toast_tuple_find_biggest_attribute() for OID8
TOAST values
This routine is used to decide if largest varlena attributes should be
compressed or not, but stuck its decisions to a hardcoded
TOAST_OID_POINTER_SIZE, which would be amiss when dealing with an oid8
TOAST value.
This information is added to the parent Relation, and can be asked on a
per-call basis of a new RelationGetToastChunkIdType().
---
src/include/utils/rel.h | 9 +++++
src/include/utils/relcache.h | 1 +
src/backend/access/common/toast_internals.c | 3 +-
src/backend/access/table/toast_helper.c | 18 ++++++++--
src/backend/utils/cache/relcache.c | 37 +++++++++++++++++++++
5 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 8bec73d55794..99d32cbcb21f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -250,6 +250,15 @@ typedef struct RelationData
*/
Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */
+ /*
+ * Type OID of the "chunk_id" column of this relation's TOAST table, i.e.
+ * OIDOID or OID8OID.
+ *
+ * This data is filled on demand by RelationGetToastChunkIdType(), not
+ * at relcache build time, so as to save in syscache lookups.
+ */
+ Oid rd_toastchunkidtype;
+
bool pgstat_enabled; /* should relation stats be counted */
/* use "struct" here to avoid needing to include pgstat.h: */
struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 89c27aa1529f..e17c84908332 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok);
extern Oid RelationGetReplicaIndex(Relation relation);
+extern Oid RelationGetToastChunkIdType(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
extern List *RelationGetDummyIndexExpressions(Relation relation);
extern List *RelationGetIndexPredicate(Relation relation);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 84c4361af23d..5a7f63253f84 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,7 +25,6 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,7 +130,7 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
- Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ Oid toast_typid = RelationGetToastChunkIdType(rel);
int32 max_chunk_size;
/* Fields that will be assembled into the TOAST pointer at the end */
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6c7257f5f94b..3235d033ea9b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -175,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
- * if not, no benefit is to be expected by compressing it.
+ * The column must be larger than the TOAST pointer that would replace it;
+ * if not, no benefit is to be expected by compressing it. Note that this
+ * choice depends on the TOAST value type, oid or oid8.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -188,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+ int32 biggest_size;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /*
+ * Size of the TOAST pointer this relation would use. A relation without
+ * a TOAST table cannot have any of its attributes moved out-of-line, but
+ * it can still have some of them compressed. Fall back to the oid size
+ * in that case.
+ */
+ if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID)
+ biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE);
+ else
+ biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index f475d7039772..65dfdbda100a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1205,6 +1205,9 @@ retry:
relation->rd_fkeylist = NIL;
relation->rd_fkeyvalid = false;
+ /* TOAST type data is not loaded till asked for */
+ relation->rd_toastchunkidtype = InvalidOid;
+
/* partitioning data is not loaded till asked for */
relation->rd_partkey = NULL;
relation->rd_partkeycxt = NULL;
@@ -5093,6 +5096,39 @@ RelationGetReplicaIndex(Relation relation)
return relation->rd_replidindex;
}
+/*
+ * RelationGetToastChunkIdType -- get the type of the relation's TOAST
+ * table "chunk_id" column
+ *
+ * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST
+ * table.
+ */
+Oid
+RelationGetToastChunkIdType(Relation relation)
+{
+ Oid toastrelid = relation->rd_rel->reltoastrelid;
+ Oid typid;
+
+ /* Quick exit if we already computed the value */
+ if (OidIsValid(relation->rd_toastchunkidtype))
+ return relation->rd_toastchunkidtype;
+
+ /* Nothing to report without a TOAST table */
+ if (!OidIsValid(toastrelid))
+ return InvalidOid;
+
+ typid = get_atttype(toastrelid, 1);
+ if (!OidIsValid(typid))
+ elog(ERROR, "cache lookup failed for TOAST relation %u",
+ toastrelid);
+ if (typid != OIDOID && typid != OID8OID)
+ elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u",
+ typid, toastrelid);
+
+ relation->rd_toastchunkidtype = typid;
+ return typid;
+}
+
/*
* RelationGetIndexExpressions -- get the index expressions for an index
*
@@ -6530,6 +6566,7 @@ load_relcache_init_file(bool shared)
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
rel->rd_droppedSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_toastchunkidtype = InvalidOid;
rel->pgstat_info = NULL;
/*
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-08 23:55 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-08 23:55 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Mon, Sep 7, 2026 at 8:50 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> There is still something that I am wondering about this part of the
> patch, though, should we still publish in the output of
> pg_control_checkpoint() and pg_controldata an Oid extracted from the
> Oid8 value? I have not done that, and that does not make the control
> file larger. Just wondering if that would be useful to keep because
> we cannot really do an oid8->oid cast, and the oid information could
> still be useful on its own?
I don't think pg_control_checkpoint() and pg_controldata emitting an
OID extracted from the OID8 value is the right idea. I understand that
tools and external modules do need to adapt to the wider value, but
that is fine for a new major version, and the control file version
bump gives them the signal to do so. Also, the extra 4 bytes from OID8
are absorbed by existing padding, so the control file stays well under
the safe size limit of 512 bytes anyway.
> The rest of the patch set is mostly following the same organization,
> and I have spent a few hours working more on the last piece, making
> cleaner the contract with the vartag handling (one could not the
> changes in varatt.h, where I have changed to inline function for the
> get/set Oid8 values in varatt_external_oid8).
>
> 0005~0007 in the set of patches cover gaps in the regression tests.
> 0005 is the same one set of SQLs as the previous patch set (mostly),
> 0006 and 0007 cover more cases with test_decoding, amcheck,
> reloptions, the compression function, and the external toast pointer
> comparison in toast_tuple_init(). They have been split as they can
> still work once the reloption supports oid8, even if we don't make the
> varatt_external added yet (last patch).
Thanks for the v15 patches. I reviewed and tested (including pgindent)
the patches.
v15-0001-Enlarge-OID-generation-to-8-bytes.patch: this looks good to
me, with one comment. Although the wraparound with an 8-byte OID is
hundreds of years away, and 0001 assumes it never happens, do we need
to handle the case where someone sets the OID counter closer to the
8-byte OID limit using pg_resetwal and still hits the wraparound? We
can either reject this in pg_resetwal, handle the wraparound case in
the caller, document it, or mention it in the commit message or
somewhere near pg_resetwal. This is not a blocker for getting 0001 in,
just to be clear. I think this needs to be fixed as part of
v15-0006-Add-support-for-oid8-TOAST-values.patch, which does not check
for an existing TOAST OID.
v15-0002-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch: this
looks good to me. For the reorderbuffer.c change, I microbenchmarked
uint32_hash and tag_hash with a small C program that calls each 200M
times on the same key value, 4-byte for uint32_hash and 8-byte for
tag_hash (the existing 4-byte chunk_id case). The per-call cost for
both is 17 to 20 ns/call (run-to-run noise), and the difference is
only about 1 to 2 ns/call. This cost may well be negligible in the
midst of hashing, lookup logic and most importantly the overall
decoding costs. So, use of tag_hash looks fine to me here.
v15-0003-Switch-pg_column_toast_chunk_id-return-value-fro.patch: this
looks good to me, with one comment. The chunk_id cast in
cluster-toast-value-reuse.spec seems redundant, because that column
comes from the output of pg_column_toast_chunk_id(). In
misc_functions.sql it is needed. This makes me think about whether we
need to support OID to OID8 comparison in more cases: existing queries
and monitoring tools that do pg_column_toast_chunk_id(x) = some_oid,
or that store the result in an OID column, will now get a
type-mismatch error and have to add an explicit cast. Could we ease
that by making the OID to OID8 cast implicit, so that an OID compared
against an OID8 is promoted automatically? This is not a blocker for
getting 0003 in, just to be clear.
v15-0004-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch: this
looks good to me, with a nit. Use
format_type_be(binary_upgrade_next_toast_chunk_id_typoid) instead of
%u for better readability in the "cannot support toast chunk_id type"
error message.
v15-0005-Add-relation-option-toast_value_type.patch: some comments.
1/ Parenthesize defaulttarg.
+/*
+ * RelationGetToastValueType
+ * Returns the relation's toast_value_type. Note multiple eval of argument!
+ */
+#define RelationGetToastValueType(relation, defaulttarg) \
+ ((relation)->rd_options ? \
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)
2/ Nit: double space after "creating".
+ <literal>chunk_id</literal> used when initially creating a toast
+ relation for this table.
3/ I understand that v15-0008 adds more tests, but can we have simple
tests in this patch as well, like creating a table with
toast_value_type='oid' and with an unknown type to verify the error?
This would make this patch independent.
4/ I checked that pg_dump emits the new value of this reloption even
when it is changed after the table was created. The existing TOAST
table and its index are not touched by the ALTER, but the new value is
stored in pg_class.reloptions, and that is what pg_dump emits. I think
this is fine, since dump and restore is the only way for a user to
migrate an existing table to an OID8 TOAST table.
5/ Following on from (4). Say I create a table with oid, insert data,
change the reloption to oid8, and then run a rewrite (VACUUM FULL,
CLUSTER or REPACK). The final table still uses oid, because the
rewrite preserves the existing TOAST table by swapping contents rather
than recreating it. But the transient table that make_new_heap()
builds does get created with the new type, because it goes through
create_toast_table(), which reads the reloption without checking
whether this is a rewrite. With oid this makes no difference, but once
oid8 is added the transient TOAST table would be oid8 while the
existing TOAST table still says oid. Is that expected?
I will get to reviewing v15-0006 to v15-0011 soon.
I verified the following cases on top of the v15 patch set.
1/ Measured the additional storage space and impact on query
performance for the same TOAST limits under 4 billion (OID limit).
At 4 GB TOAST (500k rows × 8 KB), oid8 costs +40% in TOAST index size
and +15% in heap size, with no measurable change to the TOAST table
size or full-detoast scan time. Queries [1]. The heap increase comes
from the on-disk TOAST pointer growing 4 bytes and the tuple
alignment. The TOAST index grows for the same reason on its chunk_id
key. The TOAST table size stays about the same because the extra
chunk_id bytes are small next to the roughly 2 KB of chunk_data and do
not change how many chunks fit per page.
table heap toast table toast index scan (3 reps, ms)
t_oid 25 MB 3935 MB 54 MB 3995 / 3971 / 3957
t_oid8 29 MB 3944 MB 75 MB 3996 / 3805 / 3809
delta +15.4% +0.23% +40.3% ~0 (within noise)
2/ Used oid8 for all tables by default and ran regression tests to see
if there are any issues.
make check seems happy with just one expected failure [2]. Fixing that
makes it run to the end successfully.
make check-world also seems happy with just one expected failure:
pg_amcheck 004_verify_heapam.pl fails [3] because it hardcodes a
4-byte OID TOAST pointer (HEAPTUPLE_PACK_LENGTH=58). oid8 widens the
pointer to 22 bytes, so the tuple is 62, not 58. Fixing that makes it
run to the end successfully. This makes me think we may need
pg_amcheck tests covering oid8.
I attached a diff file I used to make the tests happy on top of the
v15 patch set.
[1]
-- Server config: shared_buffers=2GB, maintenance_work_mem=1GB,
max_wal_size=8GB,
-- fsync=off, synchronous_commit=off, track_io_timing=on.
-- Release build (-O2), warm cache.
CREATE TABLE t_oid (id int, big bytea) WITH (toast_value_type='oid');
CREATE TABLE t_oid8 (id int, big bytea) WITH (toast_value_type='oid8');
ALTER TABLE t_oid ALTER COLUMN big SET STORAGE EXTERNAL;
ALTER TABLE t_oid8 ALTER COLUMN big SET STORAGE EXTERNAL;
INSERT INTO t_oid SELECT g, repeat('x',8000)::bytea FROM
generate_series(1,500000) g;
INSERT INTO t_oid8 SELECT g, repeat('x',8000)::bytea FROM
generate_series(1,500000) g;
VACUUM (ANALYZE) t_oid;
VACUUM (ANALYZE) t_oid8;
-- scan (forces full detoast through the toast index + table)
SELECT max(md5(big)) FROM t_oid;
SELECT max(md5(big)) FROM t_oid8;
[2]
# diff -U3 /local/home/rupiredd/postgres/src/test/regress/expected/psql.out
/local/home/rupiredd/postgres/src/test/regress/results/psql.out
# --- /local/home/rupiredd/postgres/src/test/regress/expected/psql.out
2026-08-20 01:24:10.447631698 +0000
# +++ /local/home/rupiredd/postgres/src/test/regress/results/psql.out
2026-09-08 23:00:17.613624770 +0000
# @@ -5193,7 +5193,7 @@
# TOAST table "pg_toast.pg_toast_2619"
# Column | Type
# ------------+---------
# - chunk_id | oid
# + chunk_id | oid8
# chunk_seq | integer
# chunk_data | bytea
# Owning table: "pg_catalog.pg_statistic"
[3]
doesn't match '(?^:(?^ms:heap table "postgres\.public\.test", block 0,
offset 5:\s+)data begins at offset 152 beyond the tuple length 58)'
# Looks like you failed 1 test of 32.
t/004_verify_heapam.pl ... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/32 subtests
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
Attachments:
nocfbot-oid8-default-experiment.diff (application/octet-stream, 2.8K) download
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 57ebaec69c7..152c8899e7a 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -166,7 +166,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
if (!needs_toast_table(rel))
return false;
- value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
+ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID8);
/* no default clause to catch new values added */
switch (value_type)
diff --git a/src/bin/pg_amcheck/t/004_verify_heapam.pl b/src/bin/pg_amcheck/t/004_verify_heapam.pl
index 95f1f34c90d..31287c52157 100644
--- a/src/bin/pg_amcheck/t/004_verify_heapam.pl
+++ b/src/bin/pg_amcheck/t/004_verify_heapam.pl
@@ -83,8 +83,8 @@ use Test::More;
# it is convenient enough to do it this way. We define packing code
# constants here, where they can be compared easily against the layout.
-use constant HEAPTUPLE_PACK_CODE => 'LLLSSSSSCCLLCCCCCCCCCCllLL';
-use constant HEAPTUPLE_PACK_LENGTH => 58; # Total size
+use constant HEAPTUPLE_PACK_CODE => 'LLLSSSSSCCLLCCCCCCCCCCllLLL';
+use constant HEAPTUPLE_PACK_LENGTH => 62; # Total size
# Read a tuple of our table from a heap page.
#
@@ -130,6 +130,7 @@ sub read_tuple
c_va_rawsize => shift,
c_va_extinfo => shift,
c_va_valueid => shift,
+ c_va_valueid_hi => shift,
c_va_toastrelid => shift);
# Stitch together the text for column 'b'
$tup{b} = join('', map { chr($tup{"b_body$_"}) } (1 .. 7));
@@ -163,7 +164,8 @@ sub write_tuple
$tup->{b_body6}, $tup->{b_body7},
$tup->{c_va_header}, $tup->{c_va_vartag},
$tup->{c_va_rawsize}, $tup->{c_va_extinfo},
- $tup->{c_va_valueid}, $tup->{c_va_toastrelid});
+ $tup->{c_va_valueid}, $tup->{c_va_valueid_hi},
+ $tup->{c_va_toastrelid});
sysseek($fh, $offset, 0)
or BAIL_OUT("sysseek failed: $!");
defined(syswrite($fh, $buffer, HEAPTUPLE_PACK_LENGTH))
@@ -496,7 +498,7 @@ for (my $tupidx = 0; $tupidx < $ROWCOUNT; $tupidx++)
$tup->{t_hoff} += 128;
push @expected,
- qr/${$header}data begins at offset 152 beyond the tuple length 58/,
+ qr/${$header}data begins at offset 152 beyond the tuple length 62/,
qr/${$header}tuple data should begin at byte 24, but actually begins at byte 152 \(3 attributes, no nulls\)/;
}
elsif ($offnum == 6)
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index e8605dd041f..24845ede6c3 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -5193,7 +5193,7 @@ drop role regress_partitioning_role;
TOAST table "pg_toast.pg_toast_2619"
Column | Type
------------+---------
- chunk_id | oid
+ chunk_id | oid8
chunk_seq | integer
chunk_data | bytea
Owning table: "pg_catalog.pg_statistic"
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-09 00:58 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 3 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-09 00:58 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Sep 08, 2026 at 04:55:16PM -0700, Bharath Rupireddy wrote:
> I don't think pg_control_checkpoint() and pg_controldata emitting an
> OID extracted from the OID8 value is the right idea. I understand that
> tools and external modules do need to adapt to the wider value, but
> that is fine for a new major version, and the control file version
> bump gives them the signal to do so. Also, the extra 4 bytes from OID8
> are absorbed by existing padding, so the control file stays well under
> the safe size limit of 512 bytes anyway.
I don't mean to extend the on-disk file with an extra 4-bytes for the
Oid. I mean to *show* an extra field based on a casted value. These
are two different things.
> v15-0001-Enlarge-OID-generation-to-8-bytes.patch: this looks good to
> me, with one comment. Although the wraparound with an 8-byte OID is
> hundreds of years away, and 0001 assumes it never happens, do we need
> to handle the case where someone sets the OID counter closer to the
> 8-byte OID limit using pg_resetwal and still hits the wraparound? We
> can either reject this in pg_resetwal, handle the wraparound case in
> the caller, document it, or mention it in the commit message or
> somewhere near pg_resetwal. This is not a blocker for getting 0001 in,
> just to be clear. I think this needs to be fixed as part of
> v15-0006-Add-support-for-oid8-TOAST-values.patch, which does not check
> for an existing TOAST OID.
A few things worth noting: we have in xlog_redo() and varsup.h a
couple of comments that mention wraparound without mentioning the oid8
bits that can bypass it. These need to be slightly adjusted..
I can fancy you with one more case, because there is no entry recheck
in the TOAST insert path when using an 8-byte OID:
- Insert some TOAST entries with toast_value_type=oid8.
- Reset to a past value with pg_resetwal.
- Reinsert, failing hard on INSERT due to a duplicated key.
My best answer is don't set the counter to the past. Somebody using
pg_resetwal is looking for trouble already. It's true that not being
able to handle the OID8 value conflicts like the normal TOAST path
could be seen as a defect of the OID8 path, but adding an index check
(while doable of course), is just wasting resources with 8-bytes,
because in practice that's never hit. So "don't set it to the past"
seems like an answer good enough? That sounds to me like a
documentation change.
> v15-0003-Switch-pg_column_toast_chunk_id-return-value-fro.patch: this
> looks good to me, with one comment. The chunk_id cast in
> cluster-toast-value-reuse.spec seems redundant, because that column
> comes from the output of pg_column_toast_chunk_id(). In
> misc_functions.sql it is needed. This makes me think about whether we
> need to support OID to OID8 comparison in more cases: existing queries
> and monitoring tools that do pg_column_toast_chunk_id(x) = some_oid,
> or that store the result in an OID column, will now get a
> type-mismatch error and have to add an explicit cast. Could we ease
> that by making the OID to OID8 cast implicit, so that an OID compared
> against an OID8 is promoted automatically? This is not a blocker for
> getting 0003 in, just to be clear.
My reply here would be to cast the some_oid to oid8, or invent a
secondary oid8 specific function, leaving pg_column_toast_chunk_id()
alone and failing if trying to use it with an oid8 TOAST table?
I suspect that there are not that many users of
pg_column_toast_chunk_id() anyway. May be wrong of course, but a cast
to oid8 is fine by me. I'm just not sure that it is worth the churn
of having a second function with a different return type.
> v15-0004-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch: this
> looks good to me, with a nit. Use
> format_type_be(binary_upgrade_next_toast_chunk_id_typoid) instead of
> %u for better readability in the "cannot support toast chunk_id type"
> error message.
Not sure it matter. That should not be reachable in practice.
> v15-0005-Add-relation-option-toast_value_type.patch: some comments.
> 1/ Parenthesize defaulttarg.
> 2/ Nit: double space after "creating".
Noted these down for later.
> 3/ I understand that v15-0008 adds more tests, but can we have simple
> tests in this patch as well, like creating a table with
> toast_value_type='oid' and with an unknown type to verify the error?
> This would make this patch independent.
Yeah, I guess that you're right to complain here. Most of the
reloption tests could be moved to the patch that introduces the
reloption, for clarity.
> 5/ Following on from (4). Say I create a table with oid, insert data,
> change the reloption to oid8, and then run a rewrite (VACUUM FULL,
> CLUSTER or REPACK). The final table still uses oid, because the
> rewrite preserves the existing TOAST table by swapping contents rather
> than recreating it. But the transient table that make_new_heap()
> builds does get created with the new type, because it goes through
> create_toast_table(), which reads the reloption without checking
> whether this is a rewrite. With oid this makes no difference, but once
> oid8 is added the transient TOAST table would be oid8 while the
> existing TOAST table still says oid. Is that expected?
I am not completely sure to follow here? Wouldn't that be the case of
create_toast_table() defining a OIDOldToast. In this case, we inherit
the type ID from the old table when creating what would be the
transient table. See the comment named as "break the world" in
create_toast_table() added in the patch. Or perhaps I'm missing your
point entirely.
> 1/ Measured the additional storage space and impact on query
> performance for the same TOAST limits under 4 billion (OID limit).
>
> At 4 GB TOAST (500k rows × 8 KB), oid8 costs +40% in TOAST index size
> and +15% in heap size, with no measurable change to the TOAST table
> size or full-detoast scan time. Queries [1]. The heap increase comes
> from the on-disk TOAST pointer growing 4 bytes and the tuple
> alignment. The TOAST index grows for the same reason on its chunk_id
> key. The TOAST table size stays about the same because the extra
> chunk_id bytes are small next to the roughly 2 KB of chunk_data and do
> not change how many chunks fit per page.
Yeah, the cost is part of the game here..
> 2/ Used oid8 for all tables by default and ran regression tests to see
> if there are any issues.
Did this one as well at some point. I'm pretty sure I've caught some
of the rewrite bugs with that, but I don't recall entirely..
> [3]
> doesn't match '(?^:(?^ms:heap table "postgres\.public\.test", block 0,
> offset 5:\s+)data begins at offset 152 beyond the tuple length 58)'
> # Looks like you failed 1 test of 32.
> t/004_verify_heapam.pl ... Dubious, test returned 1 (wstat 256, 0x100)
> Failed 1/32 subtests
This needs manipulation of on-disk data. I have done that at some
point but I did not see that as worth the extra cost. Having tests
for the valid cases sounded more than enough, because we care about
making sure that the vartags are passed around right. Whether they
actual report a corruption is less interesting.
Before reworking on a new rebase of the whole patch set, attached is
an adjusted 0001 for the 8-byte counter, with adjustments for the docs
of pg_resetwal and fixes for a few comments that worry about OID
wraparound, that I have noticed after more review.
How does this part look?
--
Michael
Attachments:
v16-0001-Enlarge-OID-generation-to-8-bytes.patch (text/plain, 24.3K) download
From f05d9278a7101075f21f7fc8f52f74f5d22d60f3 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 9 Sep 2026 09:56:25 +0900
Subject: [PATCH v16] Enlarge OID generation to 8 bytes
This change is combined with an update of the control file, with
CheckPoint.nextOid enlarged from Oid to Oid8. pg_control_checkpoint(),
pg_resetwal and pg_upgrade are updated to accomodate to this change, to
make sure that the value is not truncated due to 32-bit limitations when
used.
This commit adds a new routine called GetNewObjectId8() in varsup.c,
which is able to retrieve a 8-byte OID. GetNewObjectId() is kept
compatible with its origin, where we still check that the lower 32 bits
of the counter do not wraparound, handling the FirstNormalObjectId case.
A couple of comments in the xlog redo logic referred to wraparound
issues, which is not the case anymore with this change in place, so
refresh these.
Tests are added to cover that, across all the tools covered by this
change.
Bump catalog version.
Bump XLOG_PAGE_MAGIC, due to XLOG_NEXTOID.
---
src/include/access/transam.h | 8 +--
src/include/access/xlog.h | 2 +-
src/include/access/xlog_internal.h | 2 +-
src/include/catalog/pg_control.h | 4 +-
src/include/catalog/pg_proc.dat | 2 +-
src/backend/access/rmgrdesc/xlogdesc.c | 8 +--
src/backend/access/transam/varsup.c | 59 ++++++++++++++++-------
src/backend/access/transam/xlog.c | 26 +++++-----
src/backend/access/transam/xlogrecovery.c | 2 +-
src/backend/utils/misc/pg_controldata.c | 2 +-
src/bin/pg_controldata/pg_controldata.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 10 ++--
src/bin/pg_resetwal/t/001_basic.pl | 17 +++++++
src/bin/pg_upgrade/controldata.c | 2 +-
src/bin/pg_upgrade/pg_upgrade.c | 2 +-
src/bin/pg_upgrade/pg_upgrade.h | 2 +-
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 25 ++++++++++
doc/src/sgml/func/func-info.sgml | 2 +-
doc/src/sgml/ref/pg_resetwal.sgml | 10 ++--
19 files changed, 128 insertions(+), 59 deletions(-)
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 55a4ab26b348..2d85e4c83c2e 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -186,8 +186,9 @@ FullTransactionIdAdvance(FullTransactionId *dest)
* more painful, however, since some user-defined OIDs will appear in
* on-disk data; such a change would probably break pg_upgrade.
*
- * NOTE: if the OID generator wraps around, we skip over OIDs 0-16383
- * and resume with 16384. This minimizes the odds of OID conflict, by not
+ * NOTE: the OID generator is 8 bytes wide (see GetNewObjectId8())). For
+ * its lower 4 bytes, we skip over OIDs 0-16383 and resume with 16384.
+ * This minimizes the odds of OID conflict when using GetNewObjectId(), by not
* reassigning OIDs that might have been assigned during initdb. Critically,
* it also ensures that no user-created object will be considered pinned.
* ----------
@@ -211,7 +212,7 @@ typedef struct TransamVariablesData
/*
* These fields are protected by OidGenLock.
*/
- Oid nextOid; /* next OID to assign */
+ Oid8 nextOid; /* next OID (8 bytes) to assign */
uint32 oidCount; /* OIDs available before must do XLOG work */
/*
@@ -353,6 +354,7 @@ extern void SetTransactionIdLimit(TransactionId oldest_datfrozenxid,
extern void AdvanceOldestClogXid(TransactionId oldest_datfrozenxid);
extern bool ForceTransactionIdLimitUpdate(void);
extern Oid GetNewObjectId(void);
+extern Oid8 GetNewObjectId8(void);
extern void StopGeneratingPinnedObjectIds(void);
#ifdef USE_ASSERT_CHECKING
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 130ba9291096..7a590b7e1eaa 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -284,7 +284,7 @@ extern void ShutdownXLOG(int code, Datum arg);
extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
-extern void XLogPutNextOid(Oid nextOid);
+extern void XLogPutNextOid(Oid8 nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern XLogRecPtr XLogAssignLSN(void);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index be718993401b..bf609c3c703d 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -32,7 +32,7 @@
/*
* Each page of XLOG file has a header like this:
*/
-#define XLOG_PAGE_MAGIC 0xD121 /* can be used as WAL version indicator */
+#define XLOG_PAGE_MAGIC 0xD122 /* can be used as WAL version indicator */
typedef struct XLogPageHeaderData
{
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 7b5404460ec2..89ab43dd4fc7 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -22,7 +22,7 @@
/* Version identifier for this pg_control format */
-#define PG_CONTROL_VERSION 1903
+#define PG_CONTROL_VERSION 2000
/* Nonce key length, see below */
#define MOCK_AUTH_NONCE_LEN 32
@@ -43,7 +43,7 @@ typedef struct CheckPoint
int wal_level; /* current wal_level */
bool logicalDecodingEnabled; /* current logical decoding status */
FullTransactionId nextXid; /* next free transaction ID */
- Oid nextOid; /* next free OID */
+ Oid8 nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 960763ee50ba..c53ce68c7173 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12429,7 +12429,7 @@
descr => 'pg_controldata checkpoint state information as a function',
proname => 'pg_control_checkpoint', provolatile => 'v',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,bool,text,oid,xid,int8,xid,oid,xid,xid,oid,xid,xid,int4,timestamptz}',
+ proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,bool,text,oid8,xid,int8,xid,oid,xid,xid,oid,xid,xid,int4,timestamptz}',
proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,logical_decoding,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,data_page_checksum_version,checkpoint_time}',
prosrc => 'pg_control_checkpoint' },
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 2468a7d25785..64e749c1e6b7 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -101,7 +101,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
CheckPoint *checkpoint = (CheckPoint *) rec;
appendStringInfo(buf, "redo %X/%08X; "
- "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "
+ "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid " OID8_FORMAT "; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
@@ -129,10 +129,10 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
else if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
- memcpy(&nextOid, rec, sizeof(Oid));
- appendStringInfo(buf, "%u", nextOid);
+ memcpy(&nextOid, rec, sizeof(Oid8));
+ appendStringInfo(buf, OID8_FORMAT, nextOid);
}
else if (info == XLOG_RESTORE_POINT)
{
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index cb68d8fa974a..889bdebe1880 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -541,31 +541,51 @@ ForceTransactionIdLimitUpdate(void)
/*
- * GetNewObjectId -- allocate a new OID
+ * GetNewObjectId -- allocate a new OID (4 bytes)
*
- * OIDs are generated by a cluster-wide counter. Since they are only 32 bits
- * wide, counter wraparound will occur eventually, and therefore it is unwise
- * to assume they are unique unless precautions are taken to make them so.
- * Hence, this routine should generally not be used directly. The only direct
- * callers should be GetNewOidWithIndex() and GetNewRelFileNumber() in
- * catalog/catalog.c.
+ * OIDs are generated by a cluster-wide counter. The callers of this routine
+ * expect a 32 bit-wide counter. Counter wraparound will occur eventually,
+ * and therefore it is unwise to assume they are unique unless precautions are
+ * taken to make them so. This routine should generally not be used directly.
+ * The only direct callers should be GetNewOidWithIndex() and
+ * GetNewRelFileNumber() in catalog/catalog.c.
*/
Oid
GetNewObjectId(void)
{
- Oid result;
+ return (Oid) GetNewObjectId8();
+}
+
+/*
+ * GetNewObjectId8 -- allocate a new OID (8 bytes)
+ *
+ * This routine can be called directly if the consumer of the OID allocated
+ * stores the counter in an 8-byte space, where wraparound does not matter.
+ * We still need to care about the wraparound case in the low 32 bits of the
+ * space allocated, GetNewObjectId() expecting OIDs to never be allocated
+ * up to FirstNormalObjectId.
+ */
+Oid8
+GetNewObjectId8(void)
+{
+ Oid8 result;
+ Oid nextoid_lo;
+ uint32 nextoid_hi;
/* safety check, we should never get this far in a HS standby */
if (RecoveryInProgress())
elog(ERROR, "cannot assign OIDs during recovery");
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
+ nextoid_lo = (Oid) TransamVariables->nextOid;
+ nextoid_hi = (uint32) (TransamVariables->nextOid >> 32);
/*
- * Check for wraparound of the OID counter. We *must* not return 0
- * (InvalidOid), and in normal operation we mustn't return anything below
- * FirstNormalObjectId since that range is reserved for initdb (see
- * IsCatalogRelationOid()). Note we are relying on unsigned comparison.
+ * Check for wraparound of the OID counter in its lower 4 bytes. We *must*
+ * not return 0 (InvalidOid), and in normal operation we mustn't return
+ * anything below FirstNormalObjectId since that range is reserved for
+ * initdb (see IsCatalogRelationOid()). Note we are relying on unsigned
+ * comparison.
*
* During initdb, we start the OID generator at FirstGenbkiObjectId, so we
* only wrap if before that point when in bootstrap or standalone mode.
@@ -575,26 +595,29 @@ GetNewObjectId(void)
* available for automatic assignment during initdb, while ensuring they
* will never conflict with user-assigned OIDs.
*/
- if (TransamVariables->nextOid < ((Oid) FirstNormalObjectId))
+ if (nextoid_lo < ((Oid) FirstNormalObjectId))
{
if (IsPostmasterEnvironment)
{
/* wraparound, or first post-initdb assignment, in normal mode */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
else
{
/* we may be bootstrapping, so don't enforce the full range */
- if (TransamVariables->nextOid < ((Oid) FirstGenbkiObjectId))
+ if (nextoid_lo < ((Oid) FirstGenbkiObjectId))
{
/* wraparound in standalone mode (unlikely but possible) */
- TransamVariables->nextOid = FirstNormalObjectId;
+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
}
}
+ /* Set next OID in its 8-byte space */
+ TransamVariables->nextOid = ((Oid8) nextoid_hi) << 32 | nextoid_lo;
+
/* If we run out of logged for use oids then we must log more */
if (TransamVariables->oidCount == 0)
{
@@ -619,7 +642,7 @@ GetNewObjectId(void)
* to the specified value.
*/
static void
-SetNextObjectId(Oid nextOid)
+SetNextObjectId(Oid8 nextOid)
{
/* Safety check, this is only allowable during initdb */
if (IsPostmasterEnvironment)
@@ -629,7 +652,7 @@ SetNextObjectId(Oid nextOid)
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
if (TransamVariables->nextOid > nextOid)
- elog(ERROR, "too late to advance OID counter to %u, it is now %u",
+ elog(ERROR, "too late to advance OID counter to " OID8_FORMAT ", it is now " OID8_FORMAT,
nextOid, TransamVariables->nextOid);
TransamVariables->nextOid = nextOid;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 3203f2fd4ee2..b75cc23ff36b 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8604,10 +8604,10 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
* Write a NEXTOID log record
*/
void
-XLogPutNextOid(Oid nextOid)
+XLogPutNextOid(Oid8 nextOid)
{
XLogBeginInsert();
- XLogRegisterData(&nextOid, sizeof(Oid));
+ XLogRegisterData(&nextOid, sizeof(Oid8));
(void) XLogInsert(RM_XLOG_ID, XLOG_NEXTOID);
/*
@@ -8873,16 +8873,17 @@ xlog_redo(XLogReaderState *record)
if (info == XLOG_NEXTOID)
{
- Oid nextOid;
+ Oid8 nextOid;
/*
* We used to try to take the maximum of TransamVariables->nextOid and
- * the recorded nextOid, but that fails if the OID counter wraps
- * around. Since no OID allocation should be happening during replay
- * anyway, better to just believe the record exactly. We still take
- * OidGenLock while setting the variable, just in case.
+ * the recorded nextOid, but that failed back when the counter was 4
+ * bytes wide and could wrap around. Since no OID allocation should
+ * be happening during replay anyway, better to just believe the
+ * record exactly. We still take OidGenLock while setting the
+ * variable, just in case.
*/
- memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid));
+ memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid8));
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
TransamVariables->nextOid = nextOid;
TransamVariables->oidCount = 0;
@@ -9010,11 +9011,10 @@ xlog_redo(XLogReaderState *record)
* to track OID assignment through XLOG_NEXTOID records. The nextOid
* counter is from the start of the checkpoint and might well be stale
* compared to later XLOG_NEXTOID records. We could try to take the
- * maximum of the nextOid counter and our latest value, but since
- * there's no particular guarantee about the speed with which the OID
- * counter wraps around, that's a risky thing to do. In any case,
- * users of the nextOid counter are required to avoid assignment of
- * duplicates, so that a somewhat out-of-date value should be safe.
+ * maximum of the nextOid counter and our latest value, but there is
+ * no point in doing so: an online checkpoint records nextOid plus
+ * oidCountdue, which is never ahead of the last XLOG_NEXTOID record
+ * that replay has applied.
*/
/* Handle multixact */
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index acac97e89d38..fff8d57ac61d 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -840,7 +840,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
LSN_FORMAT_ARGS(checkPoint.redo),
wasShutdown ? "true" : "false"));
ereport(DEBUG1,
- (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: %u",
+ (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: " OID8_FORMAT,
U64FromFullTransactionId(checkPoint.nextXid),
checkPoint.nextOid)));
ereport(DEBUG1,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 9014f0953e99..1f47302fe2a7 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -124,7 +124,7 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid)));
nulls[7] = false;
- values[8] = ObjectIdGetDatum(ControlFile->checkPointCopy.nextOid);
+ values[8] = ObjectId8GetDatum(ControlFile->checkPointCopy.nextOid);
nulls[8] = false;
values[9] = TransactionIdGetDatum(ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 6fc87ed114d8..b785f7f40701 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -269,7 +269,7 @@ main(int argc, char *argv[])
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile->checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile->checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 79f3085d7696..63e4381e03f0 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -82,7 +82,7 @@ static TransactionId oldest_commit_ts_xid_val;
static TransactionId newest_commit_ts_xid_val;
static bool next_oid_given = false;
-static Oid next_oid_val;
+static Oid8 next_oid_val;
static bool mxids_given = false;
static MultiXactId next_mxid_val;
@@ -253,7 +253,7 @@ main(int argc, char *argv[])
case 'o':
errno = 0;
- next_oid_val = strtouint32_strict(optarg, &endptr, 0);
+ next_oid_val = strtouint64_strict(optarg, &endptr, 0);
if (endptr == optarg || *endptr != '\0' || errno != 0)
{
pg_log_error("invalid argument for option %s", "-o");
@@ -775,7 +775,7 @@ PrintControlValues(bool guessed)
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile.checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile.checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),
+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile.checkPointCopy.nextMulti);
@@ -861,7 +861,7 @@ PrintNewControlValues(void)
if (next_oid_given)
{
- printf(_("NextOID: %u\n"),
+ printf(_("NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
}
@@ -1236,7 +1236,7 @@ usage(void)
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
- printf(_(" -o, --next-oid=OID set next OID\n"));
+ printf(_(" -o, --next-oid=OID8 set next OID (8 bytes)\n"));
printf(_(" -O, --multixact-offset=OFFSET set next multitransaction offset\n"));
printf(_(" -u, --oldest-transaction-id=XID set oldest transaction ID\n"));
printf(_(" -x, --next-transaction-id=XID set next transaction ID\n"));
diff --git a/src/bin/pg_resetwal/t/001_basic.pl b/src/bin/pg_resetwal/t/001_basic.pl
index 6af97c0a7d99..f54e9bb42231 100644
--- a/src/bin/pg_resetwal/t/001_basic.pl
+++ b/src/bin/pg_resetwal/t/001_basic.pl
@@ -265,4 +265,21 @@ command_like(
$node->start;
ok(1, 'server started after reset');
+# Checks for 8 bytes OIDs.
+$node->stop;
+my $new_nextoid = "4295067296"; # 2^32 + 100,000
+command_ok([ 'pg_resetwal', '--next-oid' => $new_nextoid, $node->data_dir ],
+ 'runs with --next-oid value above 2^32');
+command_like(
+ [ 'pg_resetwal', '--dry-run', $node->data_dir ],
+ qr/^Latest checkpoint's NextOID: *$new_nextoid$/m,
+ '8-byte --next-oid value preserved in pg_control');
+$node->start;
+is( $node->safe_psql(
+ 'postgres',
+ "SELECT next_oid >= '$new_nextoid'::oid8 FROM pg_control_checkpoint()"
+ ),
+ 't',
+ 'new 8-byte OID reported');
+
done_testing();
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 7543a9880451..f0fa2b1689fa 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -245,7 +245,7 @@ get_control_data(ClusterInfo *cluster)
pg_fatal("%d: controldata retrieval problem", __LINE__);
p++; /* remove ':' char */
- cluster->controldata.chkpnt_nxtoid = str2uint(p);
+ cluster->controldata.chkpnt_nxtoid = str2uint64(p);
got_oid = true;
}
else if ((p = strstr(bufin, "Latest checkpoint's NextMultiXactId:")) != NULL)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 7366fd4627cc..3704c620603c 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -196,7 +196,7 @@ main(int argc, char **argv)
*/
prep_status("Setting next OID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
- "\"%s/pg_resetwal\" -o %u \"%s\"",
+ "\"%s/pg_resetwal\" -o " OID8_FORMAT " \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtoid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index c80e8fb4031c..4cb79bb780e6 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -216,7 +216,7 @@ typedef struct
char nextxlogfile[25];
uint32 chkpnt_nxtxid;
uint32 chkpnt_nxtepoch;
- uint32 chkpnt_nxtoid;
+ Oid8 chkpnt_nxtoid;
uint32 chkpnt_nxtmulti;
uint64 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index a5cfd53265d4..728fd6b0b120 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -588,6 +588,20 @@ $oldnode->start;
$oldnode->safe_psql('postgres', 'DROP DATABASE regression_invalid');
$oldnode->stop;
+# The OID counter is 8 bytes wide, check that it is carried. Older versions
+# may not support 8-byte OIDs, so skip in this case.
+my $big_next_oid = '4295067296'; # 2^32 + 100000
+if (!defined($ENV{oldinstall}))
+{
+ command_ok(
+ [
+ 'pg_resetwal',
+ '--next-oid' => $big_next_oid,
+ $oldnode->data_dir
+ ],
+ 'set an 8-byte OID counter in the old instance');
+}
+
# --check command works here, cleans up pg_upgrade_output.d.
command_ok(
[
@@ -624,6 +638,17 @@ ok( !-d $newnode->data_dir . "/pg_upgrade_output.d",
$newnode->start;
+# The 8-byte OID has been carried.
+if (!defined($ENV{oldinstall}))
+{
+ is( $newnode->safe_psql(
+ 'postgres',
+ "SELECT next_oid >= '$big_next_oid'::oid8 FROM pg_control_checkpoint()"
+ ),
+ 't',
+ '8-byte OID counter is carried over by pg_upgrade');
+}
+
# Check if there are any logs coming from pg_upgrade, that would only be
# retained on failure.
my $log_path = $newnode->data_dir . "/pg_upgrade_output.d";
diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml
index 1ff023b741c8..e56c9a22c42a 100644
--- a/doc/src/sgml/func/func-info.sgml
+++ b/doc/src/sgml/func/func-info.sgml
@@ -3433,7 +3433,7 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
<row>
<entry><structfield>next_oid</structfield></entry>
- <entry><type>oid</type></entry>
+ <entry><type>oid8</type></entry>
</row>
<row>
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index 664c525f481c..d34aa26413c0 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -286,17 +286,19 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>-o <replaceable class="parameter">oid</replaceable></option></term>
- <term><option>--next-oid=<replaceable class="parameter">oid</replaceable></option></term>
+ <term><option>-o <replaceable class="parameter">oid8</replaceable></option></term>
+ <term><option>--next-oid=<replaceable class="parameter">oid8</replaceable></option></term>
<listitem>
<para>
- Manually set the next OID.
+ Manually set the next OID (8 bytes).
</para>
<para>
There is no comparably easy way to determine a next OID that's beyond
the largest one in the database, but fortunately it is not critical to
- get the next-OID setting right.
+ get the next-OID setting right, as long as the counter is not moved
+ backwards. The counter is 8 bytes wide and never wraps around during
+ the life of a cluster.
</para>
</listitem>
</varlistentry>
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-09 14:14 Greg Burd <greg@burd.me>
parent: Michael Paquier <michael@paquier.xyz>
2 siblings, 1 reply; 72+ messages in thread
From: Greg Burd @ 2026-09-09 14:14 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hey Michael, Bharath, all,
First, I've been following this on the earlier thread and this one and I
greatly appreciate the work you're doing Michael. I think it is valuable
and necessary. I've been out most of the Summer on vacation (I finally
took one!) and so haven't had the time until yesterday to dive into
this.
So, I applied and tested the v15 set and then the adjusted v16-0001 on
top of current master. All good, tests passing.
Beyond running the suite I poked at the feature directly. pg_resetwal -o
5000000000 followed by an insert into an oid8 table gives a chunk_id of
5000000000 and reads back correctly, while the oid table next to it
wraps to 705032705. pg_column_toast_chunk_id() reports oid8 and the full
value. I read the test body and found that your pg_upgrade tests do
assert the 2^32+100000 counter carries over, thanks for the
attention to detail.
I spent most of my time trying to put numbers on the objections already
raised, since several of them were estimates.
I think that Matthias is right about the index, and I've reproduced it
on real TOAST indexes, not a standalone proxy table. Same data on both
sides, 40000 chunks: 917504 bytes with oid against 1277952 with oid8, so
+39.3%. With denser 2-chunk values it's 1810432 against 2539520,
+40.3%. I see that Bharath's independent run at 4GB of TOAST landed on
+40.3% as well, so I'd treat the 40% as settled.
However, I'd add the other half of that ratio, because I think it
changes how much weight the number should carry. The TOAST heap was
byte-identical in both of my runs, and the index is only 1.1% to 1.6% of
TOAST storage there, 1.35% in Bharath's. So the extra bytes come to
under 0.7% of the total TOAST footprint.
Robert's last concern on the previous thread lingered and so I wanted to
check because it seems untested so far in practice. The thought was that
some tuples storable today would not be storable at all with the wider
pointer. I tried, but couldn't reproduce the concern in practice. I
created a table of N text columns, all set STORAGE EXTERNAL, 3000 bytes
per value:
cols oid oid8
360 OK OK
370 OK row is too big: size 8824, maximum size 8160
450 OK too big
460 too big too big
So the measured ceiling on out-of-line attributes per row goes from
about 453 to about 366, call it 19%. Between roughly 370 and 455 such
columns you get a row that stores fine as oid and cannot be stored at
all as oid8, and there is nothing the TOAST code can do about it. Wide
all-text tables like might exist, but how common are they? This is a
hard wall not a slowdown, so maybe a sentence in the docs next to the
reloption would help?
On the main heap, the cost is alignment dependent and sometimes
free. One toasted column came out byte-identical, 52.22 bytes per row
either way, because MAXALIGN swallows the extra 4 bytes. Two toasted
columns showed it: 68.40 against 76.60 bytes per row, +12%. Bharath
measured +15.4% on his shape. Worth knowing that the answer is "it
depends on the row layout" rather than a flat 4 bytes per value.
Which brings me to the v16-0001 doc change. I think this sentence is
stronger than the code supports:
The counter is 8 bytes wide and never wraps around during the life of
a cluster.
It can wrap, using the option that paragraph documents. pg_resetwal
happily accepts -o 18446744073709551615, and it rejects 2^64 with a
clean error, so the boundary itself is handled. But set the counter to
18446744073709550000, start up, insert 3000 toasted rows into an oid8
table, and the counter rolls over: chunk_ids run from 16384 to
18446744073709551615, 2778 of the 3000 rows land on low ids, and
next_oid after a checkpoint reads 24576. So it is possible to induce
that corner case, however unlikely in practice.
The good news is the wrap is handled the way the design says: it resumes
at FirstNormalObjectId rather than 0, all 6000 rows I ended up with
detoast correctly, all chunk_ids were distinct, and verify_heapam()
reports zero corruption rows on both the toast relation and the main
table. So this isn't a data integrity problem.
It does mean the invariant the oid8 path leans on is
administrator-breakable, though. Since there's no entry recheck for
oid8, monotonicity is what guarantees uniqueness, and after a wrap you
no longer have it. Your own example is the sharp version of this: reset
to a past value and the next insert fails hard on the toast index unique
constraint. I reproduced that too and it fails cleanly so the
pre-existing row's data was intact. It is a hard error and not silent
corruption.
Two smaller things. The duplicate key error a user hits in that state
names an internal toast index and gives no hint about the cause, so if
you want one place to spend a few lines it might be an errhint
there. And the reloption docs currently describe what toast_value_type
sets but none of the tradeoffs above, nor that dump and restore is the
only way to move an existing table. Bharath's point 4 established that
pg_dump does emit the changed reloption, so the migration path works, it
just isn't written down. Nit, there's also a stray double space in
"creating a toast relation".
On the bigger question of per-relation sequences, I agree with Robert
that bottlenecking this work behind that change is probably the wrong
trade. This fixes is a live defect that users hit today.
So: +1 from me on the direction and on this implementation. The v16-0001
counter work reads fine to me apart from that one doc sentence. I'll
keep going on 0006 and 0010, which are the two I've read least carefully
so far.
Reproducers for the row-size and wraparound cases are trivial to
reconstruct from the numbers above, but happy to post scripts if useful.
best.
-greg
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-09 17:13 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
2 siblings, 0 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-09 17:13 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Tue, Sep 8, 2026 at 5:58 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Tue, Sep 08, 2026 at 04:55:16PM -0700, Bharath Rupireddy wrote:
> > I don't think pg_control_checkpoint() and pg_controldata emitting an
> > OID extracted from the OID8 value is the right idea. I understand that
> > tools and external modules do need to adapt to the wider value, but
> > that is fine for a new major version, and the control file version
> > bump gives them the signal to do so. Also, the extra 4 bytes from OID8
> > are absorbed by existing padding, so the control file stays well under
> > the safe size limit of 512 bytes anyway.
>
> I don't mean to extend the on-disk file with an extra 4-bytes for the
> Oid. I mean to *show* an extra field based on a casted value. These
> are two different things.
I meant changing ControlFile->checkPointCopy->nextOid to Oid8 in 0001
is okay, in addition to displaying it. Both are okay to me from the
external tooling perspective and also the on-disk control file size,
unless I am missing anything here.
> > v15-0001-Enlarge-OID-generation-to-8-bytes.patch: this looks good to
> > me, with one comment. Although the wraparound with an 8-byte OID is
> > hundreds of years away, and 0001 assumes it never happens, do we need
> > to handle the case where someone sets the OID counter closer to the
> > 8-byte OID limit using pg_resetwal and still hits the wraparound? We
> > can either reject this in pg_resetwal, handle the wraparound case in
> > the caller, document it, or mention it in the commit message or
> > somewhere near pg_resetwal. This is not a blocker for getting 0001 in,
> > just to be clear. I think this needs to be fixed as part of
> > v15-0006-Add-support-for-oid8-TOAST-values.patch, which does not check
> > for an existing TOAST OID.
>
> A few things worth noting: we have in xlog_redo() and varsup.h a
> couple of comments that mention wraparound without mentioning the oid8
> bits that can bypass it. These need to be slightly adjusted..
- * NOTE: if the OID generator wraps around, we skip over OIDs 0-16383
- * and resume with 16384. This minimizes the odds of OID conflict, by not
+ * NOTE: the OID generator is 8 bytes wide (see GetNewObjectId8())). For
+ * its lower 4 bytes, we skip over OIDs 0-16383 and resume with 16384.
+ * This minimizes the odds of OID conflict when using GetNewObjectId(), by not
LGTM.
> I can fancy you with one more case, because there is no entry recheck
> in the TOAST insert path when using an 8-byte OID:
> - Insert some TOAST entries with toast_value_type=oid8.
> - Reset to a past value with pg_resetwal.
> - Reinsert, failing hard on INSERT due to a duplicated key.
>
> My best answer is don't set the counter to the past. Somebody using
> pg_resetwal is looking for trouble already. It's true that not being
> able to handle the OID8 value conflicts like the normal TOAST path
> could be seen as a defect of the OID8 path, but adding an index check
> (while doable of course), is just wasting resources with 8-bytes,
> because in practice that's never hit. So "don't set it to the past"
> seems like an answer good enough? That sounds to me like a
> documentation change.
AFAIK, pg_resetwal is not used to play with OIDs on production systems
(especially setting it backwards or beyond the wraparound limits),
even in the extreme cases when the OID4 wraps around. That said, I may
be missing a legitimate use case for doing so. If one does it on a
production system and the OID8 counter goes back, we do not have
toastid_valueid_exists() checks in the callers for OID8 chunk_ids, so
it would cause duplicate key value violation errors when inserting
into the TOAST index.
I do not know the best way to address this. A doc caveat on
pg_resetwal (like the v16-0001) might be enough. Since there is a
single OID counter (now OID8, already set by --next-oid), another
option is to have pg_resetwal warn or error out when --next-oid would
move the counter backwards, though that would also affect its
legitimate use for OID4. A third option is to keep the uniqueness
probe on the OID8 path too, so assignment stays self-correcting like
the OID4 path. The retry cost that exists with OID4, where values
start overlapping as the OID space fills and the probe loop runs,
would not occur with OID8, since there are far too many unique values
to exhaust before wraparound, so the probe would return on the first
try. It would still add one index lookup per new TOAST value versus
the current no-probe path, so it is not free, and I can re-run the
benchmark I ran previously to measure that overhead if needed.
> > v15-0003-Switch-pg_column_toast_chunk_id-return-value-fro.patch: this
> > looks good to me, with one comment. The chunk_id cast in
> > cluster-toast-value-reuse.spec seems redundant, because that column
> > comes from the output of pg_column_toast_chunk_id(). In
> > misc_functions.sql it is needed. This makes me think about whether we
> > need to support OID to OID8 comparison in more cases: existing queries
> > and monitoring tools that do pg_column_toast_chunk_id(x) = some_oid,
> > or that store the result in an OID column, will now get a
> > type-mismatch error and have to add an explicit cast. Could we ease
> > that by making the OID to OID8 cast implicit, so that an OID compared
> > against an OID8 is promoted automatically? This is not a blocker for
> > getting 0003 in, just to be clear.
>
> My reply here would be to cast the some_oid to oid8, or invent a
> secondary oid8 specific function, leaving pg_column_toast_chunk_id()
> alone and failing if trying to use it with an oid8 TOAST table?
>
> I suspect that there are not that many users of
> pg_column_toast_chunk_id() anyway. May be wrong of course, but a cast
> to oid8 is fine by me. I'm just not sure that it is worth the churn
> of having a second function with a different return type.
Changing the existing function's return type to OID8 seems fine to me
instead of a new function here.
> > 5/ Following on from (4). Say I create a table with oid, insert data,
> > change the reloption to oid8, and then run a rewrite (VACUUM FULL,
> > CLUSTER or REPACK). The final table still uses oid, because the
> > rewrite preserves the existing TOAST table by swapping contents rather
> > than recreating it. But the transient table that make_new_heap()
> > builds does get created with the new type, because it goes through
> > create_toast_table(), which reads the reloption without checking
> > whether this is a rewrite. With oid this makes no difference, but once
> > oid8 is added the transient TOAST table would be oid8 while the
> > existing TOAST table still says oid. Is that expected?
>
> I am not completely sure to follow here? Wouldn't that be the case of
> create_toast_table() defining a OIDOldToast. In this case, we inherit
> the type ID from the old table when creating what would be the
> transient table. See the comment named as "break the world" in
> create_toast_table() added in the patch. Or perhaps I'm missing your
> point entirely.
You are right. The new reloption is not stored in the TOAST table's
pg_class row after ALTER TABLE. Instead, it lives on the heap table's
pg_class row. So, during a TOAST table rewrite, the old TOAST table's
existing chunk_id type (OID4 here) is used for the transient table.
This makes it work for the case of create table with OID4 chunk_id,
insert data, alter to change reloption to OID8, rewrite the table. The
transient table uses OID4 chunk_id, not OID8.
Flip this a bit. Create table with OID8 chunk_id, insert data, alter
to change reloption to OID4, rewrite the table. The transient table
uses OID8 chunk_id because the TOAST table was created with OID8. The
transient table can still store chunk_ids beyond the OID4 limit.
In short, the rewrite works fine with both OID8 and OID4 chunk_ids
across ALTER TABLE and rewrites.
> > 2/ Used oid8 for all tables by default and ran regression tests to see
> > if there are any issues.
>
> Did this one as well at some point. I'm pretty sure I've caught some
> of the rewrite bugs with that, but I don't recall entirely..
I think it is a good idea to keep running these tests with default
oid8 on one existing or separate BF member. This helps catch any
future changes in these areas by anyone, not necessarily related to
TOAST but to OID generation, pg_resetwal, pg_dump, pg_upgrade, etc.
> Before reworking on a new rebase of the whole patch set, attached is
> an adjusted 0001 for the 8-byte counter, with adjustments for the docs
> of pg_resetwal and fixes for a few comments that worry about OID
> wraparound, that I have noticed after more review.
>
> How does this part look?
The v16-0001 patch looks good to me (except the pg_resetwal comment I
made upthread). make check-world and pgindent are happy with it.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-09 17:14 Greg Burd <greg@burd.me>
parent: Greg Burd <greg@burd.me>
0 siblings, 1 reply; 72+ messages in thread
From: Greg Burd @ 2026-09-09 17:14 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Correction to my last message, since I was clearly under-caffeinated or
rushed to get it sent. Not AI/LLM-slop this time HUMAN-slop. ;-P
I wrote:
> I tried, but couldn't reproduce the concern in practice.
That should say the concern does reproduce. Robert's worry was that some
tuples storable today would not be storable at all once the TOAST pointer
gets wider, and that is what the table right below that sentence shows. At
370 text columns set STORAGE EXTERNAL, oid stores the row and oid8 fails
with "row is too big: size 8824, maximum size 8160". The ceiling on
out-of-line attributes per row drops from about 453 to about 366.
Doesn't change where I land, still +1 on the series, and I think this is
the one cost that deserves a sentence in the docs next to the reloption.
best.
-greg
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-09 23:18 Michael Paquier <michael@paquier.xyz>
parent: Greg Burd <greg@burd.me>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-09 23:18 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Sep 09, 2026 at 01:14:54PM -0400, Greg Burd wrote:
> That should say the concern does reproduce. Robert's worry was that some
> tuples storable today would not be storable at all once the TOAST pointer
> gets wider, and that is what the table right below that sentence shows. At
> 370 text columns set STORAGE EXTERNAL, oid stores the row and oid8 fails
> with "row is too big: size 8824, maximum size 8160". The ceiling on
> out-of-line attributes per row drops from about 453 to about 366.
Thanks for the exact numbers. 366 toast columns still looks like an
insane thing to do, so I am not much concerned by somebody having a
number of toast attributes in the [366,453[ range that worked with a
TOAST table based on oid, then trying to do a rewrite to a new toast
table based on oid8 to see the load of data fail suddenly.
> Doesn't change where I land, still +1 on the series, and I think this is
> the one cost that deserves a sentence in the docs next to the reloption.
Hmm. Okay. I'll think about something here.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-10 07:51 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-10 07:51 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Sep 10, 2026 at 08:18:57AM +0900, Michael Paquier wrote:
> Hmm. Okay. I'll think about something here.
Attached is a rebased v16 following cb298616463d. The patch set has
been slightly re-organized, with the pieces that seem worth
introducing first on top of the rest:
- 0001 for binary upgrades.
- 0002 for the reloption toast_value_type
- 0003 for adding more Oid8 in the TOAST APIs compared to Oid.
- 0004 for pg_column_toast_chunk_id() switched to Oid8
- 0005 for adding toast_value_type=oid8, with all the TOAST changes
not related to the new vartag.
- 0006 is the battery of tests for all the cases found during
development, grouped together: core things, amcheck, test_decoding..
- 0007 adds the new vartag_external.
- 0008 is the tweak for toast_tuple_find_biggest_attribute(), which
should perhaps be merged with 0007.
Most of the previous comments were related 0002 and the new reloption
(double whitespace, doc improvement regarding no effects after
initial creation of the TOAST table, parens in the defaultarg of
rel.h. regression tests). I've addressed all of them, as so as whole
0002 feels kind of OK as-is.
--
Michael
Attachments: download all (9 files, .tar.gz)
v16-0001-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch (text/plain, 9.3K) download
From c92b06571d034aef0671bc7dfa048cf1a7e9a04e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 10:57:59 +0900
Subject: [PATCH v16 1/8] Add support for TOAST chunk_id type in binary
upgrades
This commit adds a new function, which would set the type of a chunk_id
attribute for a TOAST table across upgrades. This piece currently works
only with chunk_id = OIDOID, but it is required in a follow-up patch
where support for chunk_id = OID8OID is supported on top of the existing
one.
---
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 ++++
src/backend/catalog/heap.c | 1 +
src/backend/catalog/toasting.c | 20 ++++++++++++++++++-
src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++++
src/bin/pg_dump/pg_dump.c | 10 +++++++++-
.../expected/spgist_name_ops.out | 6 ++++--
7 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae443859..117f3e3f7746 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -29,6 +29,7 @@ extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_index_pg_class_relfilenumber;
extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber;
+extern PGDLLIMPORT Oid binary_upgrade_next_toast_chunk_id_typoid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index c53ce68c7173..7d773be732ba 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11992,6 +11992,10 @@
proname => 'binary_upgrade_set_next_toast_pg_class_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_toast_pg_class_oid' },
+{ oid => '8219', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_toast_chunk_id_typoid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_toast_chunk_id_typoid' },
{ oid => '3589', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_set_next_pg_enum_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index a107193d7ab3..1c188b7a0ff3 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -80,6 +80,7 @@
/* Potentially set by pg_upgrade_support functions */
Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid;
Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid;
+Oid binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
RelFileNumber binary_upgrade_next_heap_pg_class_relfilenumber = InvalidRelFileNumber;
RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber = InvalidRelFileNumber;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 4aa52a4bd253..bc0a72268335 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -146,6 +146,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_chunkid_typid = OIDOID;
/*
* Is it already toasted?
@@ -184,6 +185,23 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!OidIsValid(binary_upgrade_next_toast_pg_class_oid))
return false;
+
+ /*
+ * The attribute type for chunk_id should have been set when requesting
+ * a TOAST table creation.
+ */
+ if (!OidIsValid(binary_upgrade_next_toast_chunk_id_typoid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("toast chunk_id type not set while in binary upgrade mode")));
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
+ binary_upgrade_next_toast_chunk_id_typoid)));
+
+ toast_chunkid_typid = binary_upgrade_next_toast_chunk_id_typoid;
+ binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
}
/*
@@ -205,7 +223,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,
+ toast_chunkid_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 760991f1adc2..f5017022d2e8 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -149,6 +149,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_toast_chunk_id_typoid(PG_FUNCTION_ARGS)
+{
+ Oid typoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_toast_chunk_id_typoid = typoid;
+
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7a12e03ca028..bb17a2b52400 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -104,6 +104,7 @@ typedef struct
RelFileNumber relfilenumber; /* object filenode */
Oid toast_oid; /* toast table OID */
RelFileNumber toast_relfilenumber; /* toast table filenode */
+ Oid toast_chunk_id_typoid; /* type of chunk_id attribute */
Oid toast_index_oid; /* toast table index OID */
RelFileNumber toast_index_relfilenumber; /* toast table index filenode */
} BinaryUpgradeClassOidItem;
@@ -5869,7 +5870,10 @@ collectBinaryUpgradeClassOids(Archive *fout)
const char *query;
query = "SELECT c.oid, c.relkind, c.relfilenode, c.reltoastrelid, "
- "ct.relfilenode, i.indexrelid, cti.relfilenode "
+ "ct.relfilenode, i.indexrelid, cti.relfilenode, "
+ "(SELECT a.atttypid FROM pg_attribute AS a "
+ " WHERE a.attrelid = c.reltoastrelid AND attname = 'chunk_id'::text) "
+ " AS toastchunktypid "
"FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_index i "
"ON (c.reltoastrelid = i.indrelid AND i.indisvalid) "
"LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) "
@@ -5891,6 +5895,7 @@ collectBinaryUpgradeClassOids(Archive *fout)
binaryUpgradeClassOids[i].toast_relfilenumber = atooid(PQgetvalue(res, i, 4));
binaryUpgradeClassOids[i].toast_index_oid = atooid(PQgetvalue(res, i, 5));
binaryUpgradeClassOids[i].toast_index_relfilenumber = atooid(PQgetvalue(res, i, 6));
+ binaryUpgradeClassOids[i].toast_chunk_id_typoid = atooid(PQgetvalue(res, i, 7));
}
PQclear(res);
@@ -5955,6 +5960,9 @@ binary_upgrade_set_pg_class_oids(Archive *fout,
appendPQExpBuffer(upgrade_buffer,
"SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n",
entry->toast_relfilenumber);
+ appendPQExpBuffer(upgrade_buffer,
+ "SELECT pg_catalog.binary_upgrade_set_next_toast_chunk_id_typoid('%u'::pg_catalog.oid);\n",
+ entry->toast_chunk_id_typoid);
/* every toast table has an index */
appendPQExpBuffer(upgrade_buffer,
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede2430..35e59d0cd83c 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -61,9 +61,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -110,9 +111,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.55.0
v16-0002-Add-relation-option-toast_value_type.patch (text/plain, 9.9K) download
From e1da067a047e4724194868ac9cdd141caabe8314 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 10 Sep 2026 16:08:50 +0900
Subject: [PATCH v16 2/8] Add relation option toast_value_type
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is created
initially. This parameter has no effect if a TOAST table exists, even
after it is modified later on, even on rewrites.
This can be set only to "oid" currently, and will be expanded with a
second mode later.
Note: perhaps it would make sense to introduce that only when support
for 8-byte OID values are added to TOAST, the split is here to ease
review.
---
src/include/utils/rel.h | 17 ++++++++++++++++
src/backend/access/common/reloptions.c | 21 +++++++++++++++++++
src/backend/catalog/toasting.c | 16 +++++++++++++++
src/bin/psql/tab-complete.in.c | 1 +
src/test/regress/expected/reloptions.out | 22 ++++++++++++++++++++
src/test/regress/sql/reloptions.sql | 12 +++++++++++
doc/src/sgml/ref/create_table.sgml | 26 ++++++++++++++++++++++++
7 files changed, 115 insertions(+)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 41ab4586c6b5..029d614b6f23 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -341,11 +341,20 @@ typedef enum StdRdOptIndexCleanup
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
} StdRdOptIndexCleanup;
+/* StdRdOptions->toast_value_type values */
+typedef enum StdRdOptToastValueType
+{
+ STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+} StdRdOptToastValueType;
+
typedef struct StdRdOptions
{
int32 vl_len_; /* varlena header (do not touch directly!) */
int fillfactor; /* page fill factor in percent (0..100) */
int toast_tuple_target; /* target for tuple toasting */
+ StdRdOptToastValueType toast_value_type; /* type assigned to chunk_id
+ * at toast table creation */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
int parallel_workers; /* max number of parallel workers */
@@ -370,6 +379,14 @@ typedef struct StdRdOptions
((relation)->rd_options ? \
((StdRdOptions *) (relation)->rd_options)->toast_tuple_target : (defaulttarg))
+/*
+ * RelationGetToastValueType
+ * Returns the relation's toast_value_type. Note multiple eval of argument!
+ */
+#define RelationGetToastValueType(relation, defaulttarg) \
+ ((relation)->rd_options ? \
+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : (defaulttarg))
+
/*
* RelationGetFillFactor
* Returns the relation's fillfactor. Note multiple eval of argument!
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index c3f6f45f4609..82491a31a9a4 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -548,6 +548,14 @@ static relopt_enum_elt_def viewCheckOptValues[] =
{(const char *) NULL} /* list terminator */
};
+/* values from StdRdOptToastValueType */
+static relopt_enum_elt_def StdRdOptToastValueTypes[] =
+{
+ /* no value for INVALID */
+ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {(const char *) NULL} /* list terminator */
+};
+
static relopt_enum enumRelOpts[] =
{
{
@@ -561,6 +569,17 @@ static relopt_enum enumRelOpts[] =
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")
},
+ {
+ {
+ "toast_value_type",
+ "Controls the attribute type of chunk_id at toast table creation",
+ RELOPT_KIND_HEAP,
+ ShareUpdateExclusiveLock
+ },
+ StdRdOptToastValueTypes,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ gettext_noop("Valid values are \"oid\".")
+ },
{
{
"buffering",
@@ -2078,6 +2097,8 @@ static const relopt_parse_elt stdRdOptionsTab[] = {
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_analyze_min_duration)},
{"toast_tuple_target", RELOPT_TYPE_INT,
offsetof(StdRdOptions, toast_tuple_target)},
+ {"toast_value_type", RELOPT_TYPE_ENUM,
+ offsetof(StdRdOptions, toast_value_type)},
{"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL,
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)},
{"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL,
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index bc0a72268335..c3f70fcd8ab1 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -159,9 +159,25 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!IsBinaryUpgrade)
{
+ StdRdOptToastValueType value_type;
+
/* Normal mode, normal check */
if (!needs_toast_table(rel))
return false;
+
+ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
+
+ /* no default clause to catch new values added */
+ switch (value_type)
+ {
+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID:
+ toast_chunkid_typid = OIDOID;
+ break;
+ case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID:
+ elog(ERROR, "unexpected toast_value_type value %d",
+ value_type);
+ break;
+ }
}
else
{
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 8e1a1d69740c..84a129efff54 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -1441,6 +1441,7 @@ static const char *const table_storage_parameters[] = {
"toast.vacuum_max_eager_freeze_failure_rate",
"toast.vacuum_truncate",
"toast_tuple_target",
+ "toast_value_type",
"user_catalog_table",
"vacuum_index_cleanup",
"vacuum_max_eager_freeze_failure_rate",
diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out
index e3a974f26112..9b4c8587aa49 100644
--- a/src/test/regress/expected/reloptions.out
+++ b/src/test/regress/expected/reloptions.out
@@ -194,6 +194,28 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid;
-- Fail on non-existent options in toast namespace
CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42);
ERROR: unrecognized parameter "not_existing_option"
+-- Test toast_value_type.
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
+ERROR: unrecognized parameter "toast_value_type"
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
+ERROR: invalid value for enum option "toast_value_type": int8
+DETAIL: Valid values are "oid".
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
+SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+ reloptions
+------------------------
+ {toast_value_type=oid}
+(1 row)
+
+DROP TABLE reloptions_test2;
+-- relkinds not supported.
+CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
+ WITH (toast_value_type = 'oid');
+ERROR: unrecognized parameter "toast_value_type"
+CREATE TABLE reloptions_test2 (i int) PARTITION BY RANGE (i)
+ WITH (toast_value_type = 'oid');
+ERROR: cannot specify storage parameters for a partitioned table
+HINT: Specify storage parameters for its leaf partitions instead.
-- Mix TOAST & heap
DROP TABLE reloptions_test;
CREATE TABLE reloptions_test (s VARCHAR) WITH
diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql
index 680c8bf86148..503d15d16e51 100644
--- a/src/test/regress/sql/reloptions.sql
+++ b/src/test/regress/sql/reloptions.sql
@@ -112,6 +112,18 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid;
-- Fail on non-existent options in toast namespace
CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42);
+-- Test toast_value_type.
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
+SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+DROP TABLE reloptions_test2;
+-- relkinds not supported.
+CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
+ WITH (toast_value_type = 'oid');
+CREATE TABLE reloptions_test2 (i int) PARTITION BY RANGE (i)
+ WITH (toast_value_type = 'oid');
+
-- Mix TOAST & heap
DROP TABLE reloptions_test;
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index e342585c7f08..79622c1a23d6 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1653,6 +1653,32 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry id="reloption-toast-value-type" xreflabel="toast_value_type">
+ <term><literal>toast_value_type</literal> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>toast_value_type</varname> storage parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the attribute type of <literal>chunk_id</literal> to use when
+ creating a <acronym>TOAST</acronym> relation for this table. The
+ default is <literal>oid</literal>.
+ This parameter cannot be set for <acronym>TOAST</acronym> tables.
+ </para>
+ <para>
+ The value is only consulted when the <acronym>TOAST</acronym> relation
+ is created. Changing it afterwards has no effect on an existing
+ <acronym>TOAST</acronym> relation, and a table rewrite such as
+ <link linkend="sql-cluster"><command>CLUSTER</command></link> or
+ <link linkend="sql-vacuum"><command>VACUUM FULL</command></link>
+ preserves the type already in use. A dump and restore, on the other
+ hand, recreates the <acronym>TOAST</acronym> relation and therefore
+ applies the current value of this parameter.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="reloption-parallel-workers" xreflabel="parallel_workers">
<term><literal>parallel_workers</literal> (<type>integer</type>)
<indexterm>
--
2.55.0
v16-0003-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch (text/plain, 16.1K) download
From ebbad9dda1f5016beb68b2f80253b27495c1242c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 13:40:41 +0900
Subject: [PATCH v16 3/8] Refactor some TOAST value ID code to use Oid8 instead
of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 23 +++++---
contrib/amcheck/verify_heapam.c | 55 +++++++++++--------
6 files changed, 60 insertions(+), 44 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index b2346fd75bd4..aaae6e0fef69 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index ff03a2b816fc..ebec31a22e40 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1987,7 +1987,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 0f234661bfa9..ca5ed8dfabca 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);
-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)
+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)
+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 3b7c658a90e6..81154c17376c 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -729,7 +729,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",
+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -741,13 +741,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -756,7 +756,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",
+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -785,7 +785,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",
+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 457d544eec93..dd4b6e8f7f45 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */
+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -5010,7 +5010,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5034,8 +5034,10 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;
+ Oid8 chunk_id;
int32 chunk_seq;
+ Oid valueid_type;
+ Datum valueid_datum;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
@@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));
+ valueid_type = TupleDescAttr(desc, 0)->atttypid;
+ valueid_datum = fastgetattr(newtup, 1, desc, &isnull);
+ if (valueid_type == OID8OID)
+ chunk_id = DatumGetObjectId8(valueid_datum);
+ else
+ chunk_id = DatumGetObjectId(valueid_datum);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
@@ -5061,11 +5068,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5174,6 +5181,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5194,13 +5202,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,
+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index bbaaec89f8ae..33b00fa4dc5c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1564,9 +1564,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 chunksize;
int32 expected_size;
int32 max_chunk_size;
+ Oid8 toast_valueid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1574,16 +1576,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",
+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",
+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1594,8 +1596,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",
+ toast_valueid,
chunk_seq));
return;
}
@@ -1614,8 +1616,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",
+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1626,8 +1628,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",
+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1637,8 +1639,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",
+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1669,6 +1671,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external_oid toast_pointer;
@@ -1777,12 +1780,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",
- toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1809,16 +1813,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",
- toast_pointer.va_valueid, cmid));
+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",
+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",
+ toast_pointer_valueid));
return true;
}
@@ -1826,8 +1830,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",
+ toast_pointer_valueid));
return true;
}
@@ -1873,6 +1877,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1903,14 +1908,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;
+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " not found in toast table",
+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",
+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.55.0
v16-0004-Switch-pg_column_toast_chunk_id-return-value-fro.patch (text/plain, 5.7K) download
From da17a3acbb615503847ed5abd87b83caaa98450e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v16 4/8] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7d773be732ba..02602d96a2be 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7907,7 +7907,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',
+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index f6a41e709ae0..4533dac11a6d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);
+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out
index cb14ddcee34b..21859fc0dce2 100644
--- a/src/test/isolation/expected/cluster-toast-value-reuse.out
+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out
@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec
index 9a2d10600b39..96cf899db296 100644
--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec
+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec
@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
index fa76bcf28ef9..cc8f034d0167 100644
--- a/src/test/modules/injection_points/specs/repack_toast.spec
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);
+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2990e0c4f286..35d6ee2ce9eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 950d9ab1a4a1..94b91964c046 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 54eeb42e5bce..70d762e73328 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>
+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.55.0
v16-0005-Add-support-for-oid8-TOAST-values.patch (text/plain, 20.3K) download
From 07cd40bd50641ed505887dd31e2b68c62b6bf9e7 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 10 Sep 2026 16:20:22 +0900
Subject: [PATCH v16 5/8] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, unfortunately casted to OID for now, as we do not have
a vartag_external able to store Oid8 values yet.
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 3 +-
src/backend/access/common/toast_internals.c | 94 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 19 ++++-
src/backend/catalog/toasting.c | 25 +++++-
src/test/regress/expected/reloptions.out | 15 +++-
src/test/regress/sql/reloptions.sql | 8 ++
doc/src/sgml/ref/create_table.sgml | 5 +-
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 22 +++--
11 files changed, 158 insertions(+), 44 deletions(-)
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 029d614b6f23..8bec73d55794 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 82491a31a9a4..ea9a04179090 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
@@ -578,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")
+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..2c180491c2ff 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,8 +132,10 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -200,24 +203,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OIDOID)
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ Assert(false);
}
else
{
@@ -263,17 +274,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OIDOID)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
+ else if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
}
}
@@ -303,7 +319,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OIDOID)
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ else if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -384,6 +403,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -395,6 +415,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -405,10 +427,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +485,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +493,24 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..a93e98788da7 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +658,18 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index c3f70fcd8ab1..57ebaec69c78 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -173,6 +174,9 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
case STDRD_OPTION_TOAST_VALUE_TYPE_OID:
toast_chunkid_typid = OIDOID;
break;
+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID8:
+ toast_chunkid_typid = OID8OID;
+ break;
case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID:
elog(ERROR, "unexpected toast_value_type value %d",
value_type);
@@ -210,7 +214,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -235,6 +240,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -353,7 +371,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OIDOID)
+ opclassIds[0] = OID_BTREE_OPS_OID;
+ else if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out
index 9b4c8587aa49..95e4528f7793 100644
--- a/src/test/regress/expected/reloptions.out
+++ b/src/test/regress/expected/reloptions.out
@@ -199,7 +199,7 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
ERROR: unrecognized parameter "toast_value_type"
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
ERROR: invalid value for enum option "toast_value_type": int8
-DETAIL: Valid values are "oid".
+DETAIL: Valid values are "oid" and "oid8".
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
reloptions
@@ -207,6 +207,19 @@ SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
{toast_value_type=oid}
(1 row)
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ chunk_id_type
+---------------
+ oid
+(1 row)
+
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
DROP TABLE reloptions_test2;
-- relkinds not supported.
CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql
index 503d15d16e51..80939ee8b5bd 100644
--- a/src/test/regress/sql/reloptions.sql
+++ b/src/test/regress/sql/reloptions.sql
@@ -117,6 +117,14 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
DROP TABLE reloptions_test2;
-- relkinds not supported.
CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 79622c1a23d6..6f9a87ef449f 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1663,7 +1663,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<para>
Specifies the attribute type of <literal>chunk_id</literal> to use when
creating a <acronym>TOAST</acronym> relation for this table. The
- default is <literal>oid</literal>.
+ default is <literal>oid</literal>; <literal>oid8</literal> assigns
+ <type>oid8</type> instead, which allows a larger number of distinct
+ <acronym>TOAST</acronym>ed values at the price of four extra bytes per
+ out-of-line pointer.
This parameter cannot be set for <acronym>TOAST</acronym> tables.
</para>
<para>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 90aae3defcbc..21bd67c11ea0 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an OID8 identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 33b00fa4dc5c..42f409548e1d 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1876,8 +1876,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1885,10 +1889,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ if (toast_typid == OIDOID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ else if (toast_typid == OID8OID)
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ else
+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v16-0006-Add-battery-of-tests-related-oid8.patch (text/plain, 37.4K) download
From 236f13cf7a86643924cec95b552d1949be69a65e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 10 Sep 2026 16:24:52 +0900
Subject: [PATCH v16 6/8] Add battery of tests related oid8
The tests added by this commit cover a large ground in terms of the
introduction of oid8 as external TOAST pointer, with the following areas
covered, as found during development of the feature:
- Tables with oid8 TOAST tables in general, under strings.sql.
- Relation rewrites do not affect the TOAST relation after initial
creation.
- oid8 TOAST with pg_column_compression().
- test_decoding, for external and extended TOAST data with oid8
pointers.
- amcheck, with oid8 TOAST relations, external and extended formats with
heap verified.
- UPDATE with out-of-line datum that belongs to another TOAST table.
This tracks a bug in toast_tuple_init(), where external vartags could
be mixed once multiple types are introduced. That's something that
absan would trigger.
These tests are added before the commit that has adds the new external
vartag, just because it is what it is.
---
src/test/regress/expected/compression.out | 41 +++
src/test/regress/expected/strings.out | 300 +++++++++++++++++++---
src/test/regress/sql/compression.sql | 22 ++
src/test/regress/sql/strings.sql | 181 ++++++++++---
contrib/amcheck/expected/check_heap.out | 40 +++
contrib/amcheck/sql/check_heap.sql | 27 ++
contrib/test_decoding/expected/toast.out | 39 +++
contrib/test_decoding/sql/toast.sql | 20 ++
8 files changed, 595 insertions(+), 75 deletions(-)
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 09f198149aa4..e9128193515a 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
(1 row)
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+ uncompressed
+--------------
+ t
+(1 row)
+
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+ uncompressed
+--------------
+ t
+(1 row)
+
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+DROP TABLE toastcomp_oid, toastcomp_oid8;
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 313c50394650..7e3a641259c2 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2105,7 +2198,140 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index ce5ea37a660c..229143f06ddd 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2;
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+DROP TABLE toastcomp_oid, toastcomp_oid8;
+
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 38946e8954df..4d6a1e02359c 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,194 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index 979e5e84e723..8686fba712d0 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -199,6 +199,44 @@ SELECT * FROM verify_heapam('test_partition',
-------+--------+--------+-----
(0 rows)
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+ relname | chunk_id_type
+-----------------+---------------
+ test_toast_oid | oid
+ test_toast_oid8 | oid8
+(2 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -232,6 +270,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 1745bae634e5..eb46fe710351 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -112,6 +112,31 @@ SELECT * FROM verify_heapam('test_partition',
startblock := NULL,
endblock := NULL);
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -139,6 +164,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
endblock := NULL);
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.out
index a757e7dc8d54..8bef6cc32af4 100644
--- a/contrib/test_decoding/expected/toast.out
+++ b/contrib/test_decoding/expected/toast.out
@@ -382,6 +382,45 @@ SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot',
COMMIT
(4 rows)
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ shortened | len
+------------------------------------------------------------------------------------+--------
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:1 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:2 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: old-key: id[integer]:1 ne..nchanged-toast-datum | 116
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: id[integer]:2 data[text]:..bcdefghijabcdefghij' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: DELETE: id[integer]:11 | 49
+ table public.toasted_oid8: DELETE: id[integer]:2 | 48
+ COMMIT | 6
+(16 rows)
+
+DROP TABLE toasted_oid8;
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sql
index d1c560a174d6..8a2c49a5c9a7 100644
--- a/contrib/test_decoding/sql/toast.sql
+++ b/contrib/test_decoding/sql/toast.sql
@@ -324,4 +324,24 @@ INSERT INTO tbl2 VALUES(1);
commit;
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+DROP TABLE toasted_oid8;
+
SELECT pg_drop_replication_slot('regression_slot');
--
2.55.0
v16-0007-Add-support-for-TOAST-pointers-as-oid8.patch (text/plain, 47.4K) download
From 59f5ed6fb9fed6dcb0770ead1224e1899ff967e1 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:48:29 +0900
Subject: [PATCH v16 7/8] Add support for TOAST pointers as oid8
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 3 +
src/include/access/heaptoast.h | 16 +-
src/include/varatt.h | 93 ++++-
src/backend/access/common/detoast.c | 151 ++++++--
src/backend/access/common/toast_compression.c | 11 +-
src/backend/access/common/toast_internals.c | 353 ++++++++++++------
src/backend/access/heap/heaptoast.c | 17 +-
src/backend/access/table/toast_helper.c | 4 +
.../replication/logical/reorderbuffer.c | 45 ++-
src/backend/utils/adt/varlena.c | 19 +-
doc/src/sgml/storage.sgml | 10 +-
contrib/amcheck/verify_heapam.c | 90 +++--
12 files changed, 598 insertions(+), 214 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..f065ea034e98 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -30,6 +30,9 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..b7ef69a8f56b 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,13 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 33979bbaaad1..1349a54371db 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -38,6 +38,44 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except for the value ID, which is 8 bytes wide. The value ID is split
+ * into two uint32 to force alignment.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==
+ sizeof(varatt_external_oid8),
+ "varatt_external_oid8 should have no padding");
+
+/* Get/set the 64-bit value ID from the lo/hi halves */
+static inline Oid8
+VARATT_EXTERNAL_OID8_GET_VALUEID(varatt_external_oid8 toast_pointer)
+{
+ return ((Oid8) toast_pointer.va_valueid_lo) |
+ (((Oid8) toast_pointer.va_valueid_hi) << 32);
+}
+
+static inline void
+VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id)
+{
+ toast_pointer->va_valueid_lo = (uint32) id;
+ toast_pointer->va_valueid_hi = (uint32) (id >> 32);
+}
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -87,6 +125,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -108,6 +147,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -360,10 +401,29 @@ VARATT_IS_EXTERNAL(const void *PTR)
/* Is varlena datum a pointer to on-disk toasted data? */
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
+{
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
+}
+
+/* Is varlena datum a pointer to on-disk toasted data with 4-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID(const void *PTR)
{
return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */
+static inline bool
+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)
+{
+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;
+}
+
/* Is varlena datum an indirect pointer? */
static inline bool
VARATT_IS_EXTERNAL_INDIRECT(const void *PTR)
@@ -519,16 +579,6 @@ VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
* Testing whether an externally-stored value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -543,4 +593,27 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid8.
+ */
+static inline Size
+VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+}
+
+static inline uint32
+VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)
+{
+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+}
+
+static inline bool
+VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)
+{
+ return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <
+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+}
+
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..393135f1b9f3 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,12 +225,32 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /* Extract extsize and compression info from the appropriate pointer */
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +260,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,14 +271,9 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,23 +359,47 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- result = (varlena *) palloc(attrsize + VARHDRSZ);
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
+
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
+ else
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
+ }
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -369,10 +408,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +437,43 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toastrelid = toast_pointer.va_toastrelid;
+ valueid = toast_pointer.va_valueid;
+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
+ }
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +486,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +499,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +508,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +609,20 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_rawsize;
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +679,20 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 3b1bd6519261..200cee6be705 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -260,7 +260,16 @@ toast_get_compression_id(varlena *attr)
* the external toast pointer. If compressed inline, fetch it from the
* toast compression header.
*/
- if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
+ cmid = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
+ }
+ else if (VARATT_IS_EXTERNAL_ONDISK_OID(attr))
{
varatt_external_oid toast_pointer;
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 2c180491c2ff..84c4361af23d 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -125,7 +125,6 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -133,9 +132,17 @@ toast_save_datum(Relation rel, Datum value,
int num_indexes;
int validIndex;
Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
Assert(OidIsValid(toast_typid));
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -165,28 +172,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -198,13 +209,14 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or OID8 TOAST relations.
+ * Choose a new value to use as the value ID for this toast value, and
+ * determine the maximum chunk size based on the TOAST table's chunk_id
+ * type.
*
* Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
@@ -213,84 +225,150 @@ toast_save_datum(Relation rel, Datum value,
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * conflict with either new or existing toast value IDs. For Oid8 tables,
+ * value conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OIDOID)
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
+ if (!OidIsValid(rel->rd_toastoid))
+ {
+ /* normal case: just choose a new Oid8 */
+ va_valueid = GetNewObjectId8();
+ }
else
- Assert(false);
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid8;
+ if (oldexternal != NULL)
+ {
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+
+ /*
+ * Only consider the old pointer if its vartag matches the
+ * kind of pointer we are about to build.
+ */
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(oldexternal))
+ {
+ varatt_external_oid8 old_toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /*
+ * This value came from the old toast table; reuse its
+ * ID.
+ */
+ va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);
+
+ /*
+ * There is a corner case here: the table rewrite
+ * might have to copy both live and recently-dead
+ * versions of a row, and those versions could easily
+ * reference the same toast value. When we copy the
+ * second or later version of such a row, reusing the
+ * OID will mean we select an OID that's already in the
+ * new toast table. Check for that, and if so, just
+ * fall through without writing the data again.
+ *
+ * While annoying and ugly-looking, this is a good
+ * thing because it ensures that we wind up with only
+ * one copy of the toast value when there is only one
+ * copy in the old toast table. Before we detected
+ * this case, we'd have made multiple copies, wasting
+ * space; and what's worse, the copies belonging to
+ * already-deleted heap tuples would not be reclaimed
+ * by VACUUM.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /*
+ * Match, so short-circuit the data storage loop
+ * below.
+ */
+ data_todo = 0;
+ }
+ }
+ }
+ }
+ if (va_valueid == InvalidOid8)
+ {
+ /*
+ * New value. For Oid8 we just pick a new ID without worrying
+ * about conflicts.
+ */
+ va_valueid = GetNewObjectId8();
+ }
+ }
+
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
+ if (!OidIsValid(rel->rd_toastoid))
{
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ /* normal case: just choose an unused OID */
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
+ else
+ {
+ /* rewrite case: check to see if value was in old toast table */
+ va_valueid = InvalidOid;
+ if (oldexternal != NULL)
{
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
+ /* Only reuse a pointer of the matching type; same as above */
+ if (VARATT_IS_EXTERNAL_ONDISK_OID(oldexternal))
{
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
+ varatt_external_oid old_toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
+ {
+ /*
+ * This value came from the old toast table; reuse its
+ * OID.
+ */
+ va_valueid = old_toast_pointer.va_valueid;
+
+ /*
+ * Corner case during table rewrite with multiple
+ * versions of the same row. See above for details.
+ */
+ if (toastrel_valueid_exists(toastrel, va_valueid))
+ {
+ /*
+ * Match, so short-circuit the data storage loop
+ * below.
+ */
+ data_todo = 0;
+ }
+ }
}
}
- }
- if (toast_pointer.va_valueid == InvalidOid)
- {
- /*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
- */
- if (toast_typid == OIDOID)
+ if (va_valueid == InvalidOid)
{
+ /*
+ * new value; must choose a value that doesn't conflict in
+ * either old or new toast table.
+ */
do
{
- toast_pointer.va_valueid =
+ va_valueid =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ va_valueid));
}
- else if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
}
+
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
/*
@@ -314,15 +392,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
- if (toast_typid == OIDOID)
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
- else if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -376,11 +454,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(&toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -395,7 +496,6 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -403,42 +503,75 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toast_typid;
+ Oid toastrelid;
+ vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
/*
- * Open the toast relation and its indexes
+ * Determine the pointer type from the datum's vartag and extract the
+ * toast relation OID and value ID accordingly. The vartag tells us
+ * everything we need - no TOAST table schema lookup required.
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
- toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ tag = VARTAG_EXTERNAL(attr);
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
+ if (tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- if (toast_typid == OIDOID)
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toastrelid = toast_pointer8.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ Assert(tag == VARTAG_ONDISK_OID);
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toastrelid = toast_pointer.va_toastrelid;
+
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toastrelid, RowExclusiveLock);
+
+ /* Fetch valid relation used for process */
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
+
+ /*
+ * Setup a scan key to find chunks with matching va_valueid
+ */
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(toast_pointer.va_valueid));
- else
- Assert(false);
+ }
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -499,18 +632,16 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
- else if (toast_typid == OID8OID)
+ if (toast_typid == OID8OID)
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
ObjectId8GetDatum(valueid));
else
- Assert(false);
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index a93e98788da7..31e6115de201 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -650,7 +650,10 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ if (toast_typid == OID8OID)
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+ else
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
@@ -658,18 +661,16 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
- else if (toast_typid == OID8OID)
+ if (toast_typid == OID8OID)
ScanKeyInit(&toastkey[0],
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
ObjectId8GetDatum(valueid));
else
- Assert(false);
+ ScanKeyInit(&toastkey[0],
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6f2d691cc681..6c7257f5f94b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc)
/*
* If the old value is stored on disk, check if it has changed so
* we have to delete it later.
+ *
+ * Note that TOAST pointers could have different vartags, for oid
+ * or oid8, and these can have a different sizes.
*/
if (att->attlen == -1 && !ttc->ttc_oldisnull[i] &&
VARATT_IS_EXTERNAL_ONDISK(old_value))
{
if (ttc->ttc_isnull[i] ||
!VARATT_IS_EXTERNAL_ONDISK(new_value) ||
+ VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) ||
memcmp(old_value, new_value,
VARSIZE_EXTERNAL(old_value)) != 0)
{
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dd4b6e8f7f45..086ef31e4f2f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5176,12 +5176,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* va_rawsize is the size of the original datum -- including header */
varatt_external_oid toast_pointer;
+ varatt_external_oid8 toast_pointer8;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
+ int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5198,11 +5200,21 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena_pointer = (varlena *) DatumGetPointer(attrs[natt]);
/* no need to do anything if the tuple isn't external */
- if (!VARATT_IS_EXTERNAL(varlena_pointer))
+ if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);
+ toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ rawsize = toast_pointer8.va_rawsize;
+ }
+ else
+ {
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
+ rawsize = toast_pointer.va_rawsize;
+ }
/*
* Check whether the toast tuple changed, replace if so.
@@ -5220,7 +5232,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(rawsize);
ent->reconstructed = reconstructed;
@@ -5245,13 +5257,28 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ /* Verify size and set compression status based on pointer type */
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ {
+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
+ else
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
+ }
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 4533dac11a6d..0616db775558 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,7 +4261,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external_oid toast_pointer;
+ Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4288,9 +4288,22 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ {
+ varatt_external_oid8 toast_pointer;
- PG_RETURN_OID8(toast_pointer.va_valueid);
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ result = toast_pointer.va_valueid;
+ }
+
+ PG_RETURN_OID8(result);
}
/*
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 21bd67c11ea0..3561a6a67ece 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,7 +417,9 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the
+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 42f409548e1d..86dc9cf51a93 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -75,7 +75,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1566,9 +1568,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 max_chunk_size;
Oid8 toast_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ toast_valueid = ta->va_valueid;
+
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1672,8 +1677,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1732,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1777,26 +1784,46 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
/*
- * Must copy attr into toast_pointer for alignment considerations
+ * Must copy attr into toast_pointer for alignment considerations.
+ * Branch on the tag to determine which pointer type to extract.
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ va_tag_value = VARTAG_EXTERNAL(attr);
+ if (va_tag_value == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer8;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
+ toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
+ va_rawsize = toast_pointer8.va_rawsize;
+ va_extinfo = toast_pointer8.va_extinfo;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ /* Must copy to access aligned fields */
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
+ va_rawsize = toast_pointer.va_rawsize;
+ va_extinfo = toast_pointer.va_extinfo;
+ }
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1850,7 +1877,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ /* The pointer has already been decoded above, just reuse it */
+ ta->tag = va_tag_value;
+ ta->va_valueid = toast_pointer_valueid;
+ ta->va_extinfo = va_extinfo;
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1878,29 +1908,33 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
int32 max_chunk_size;
Oid8 toast_valueid;
- Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- if (toast_typid == OIDOID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)
+ if (ta->tag == VARTAG_ONDISK_OID8)
+ {
+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
+
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(ta->toast_pointer.va_valueid));
+ ObjectId8GetDatum(toast_valueid));
+ }
else
- Assert(false);
+ {
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+
+ ScanKeyInit(&toastkey,
+ (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) toast_valueid));
+ }
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1920,8 +1954,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
--
2.55.0
v16-0008-Fix-toast_tuple_find_biggest_attribute-for-OID8-.patch (text/plain, 6.7K) download
From a1cb61b0049a34f6c78180920a80458c47d2a1c4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 15:32:34 +0900
Subject: [PATCH v16 8/8] Fix toast_tuple_find_biggest_attribute() for OID8
TOAST values
This routine is used to decide if largest varlena attributes should be
compressed or not, but stuck its decisions to a hardcoded
TOAST_OID_POINTER_SIZE, which would be amiss when dealing with an oid8
TOAST value.
This information is added to the parent Relation, and can be asked on a
per-call basis of a new RelationGetToastChunkIdType().
---
src/include/utils/rel.h | 9 +++++
src/include/utils/relcache.h | 1 +
src/backend/access/common/toast_internals.c | 3 +-
src/backend/access/table/toast_helper.c | 18 ++++++++--
src/backend/utils/cache/relcache.c | 37 +++++++++++++++++++++
5 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 8bec73d55794..99d32cbcb21f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -250,6 +250,15 @@ typedef struct RelationData
*/
Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */
+ /*
+ * Type OID of the "chunk_id" column of this relation's TOAST table, i.e.
+ * OIDOID or OID8OID.
+ *
+ * This data is filled on demand by RelationGetToastChunkIdType(), not
+ * at relcache build time, so as to save in syscache lookups.
+ */
+ Oid rd_toastchunkidtype;
+
bool pgstat_enabled; /* should relation stats be counted */
/* use "struct" here to avoid needing to include pgstat.h: */
struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 89c27aa1529f..e17c84908332 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok);
extern Oid RelationGetReplicaIndex(Relation relation);
+extern Oid RelationGetToastChunkIdType(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
extern List *RelationGetDummyIndexExpressions(Relation relation);
extern List *RelationGetIndexPredicate(Relation relation);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 84c4361af23d..5a7f63253f84 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,7 +25,6 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,7 +130,7 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
- Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ Oid toast_typid = RelationGetToastChunkIdType(rel);
int32 max_chunk_size;
/* Fields that will be assembled into the TOAST pointer at the end */
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6c7257f5f94b..3235d033ea9b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -175,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
- * if not, no benefit is to be expected by compressing it.
+ * The column must be larger than the TOAST pointer that would replace it;
+ * if not, no benefit is to be expected by compressing it. Note that this
+ * choice depends on the TOAST value type, oid or oid8.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -188,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+ int32 biggest_size;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /*
+ * Size of the TOAST pointer this relation would use. A relation without
+ * a TOAST table cannot have any of its attributes moved out-of-line, but
+ * it can still have some of them compressed. Fall back to the oid size
+ * in that case.
+ */
+ if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID)
+ biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE);
+ else
+ biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index f475d7039772..65dfdbda100a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1205,6 +1205,9 @@ retry:
relation->rd_fkeylist = NIL;
relation->rd_fkeyvalid = false;
+ /* TOAST type data is not loaded till asked for */
+ relation->rd_toastchunkidtype = InvalidOid;
+
/* partitioning data is not loaded till asked for */
relation->rd_partkey = NULL;
relation->rd_partkeycxt = NULL;
@@ -5093,6 +5096,39 @@ RelationGetReplicaIndex(Relation relation)
return relation->rd_replidindex;
}
+/*
+ * RelationGetToastChunkIdType -- get the type of the relation's TOAST
+ * table "chunk_id" column
+ *
+ * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST
+ * table.
+ */
+Oid
+RelationGetToastChunkIdType(Relation relation)
+{
+ Oid toastrelid = relation->rd_rel->reltoastrelid;
+ Oid typid;
+
+ /* Quick exit if we already computed the value */
+ if (OidIsValid(relation->rd_toastchunkidtype))
+ return relation->rd_toastchunkidtype;
+
+ /* Nothing to report without a TOAST table */
+ if (!OidIsValid(toastrelid))
+ return InvalidOid;
+
+ typid = get_atttype(toastrelid, 1);
+ if (!OidIsValid(typid))
+ elog(ERROR, "cache lookup failed for TOAST relation %u",
+ toastrelid);
+ if (typid != OIDOID && typid != OID8OID)
+ elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u",
+ typid, toastrelid);
+
+ relation->rd_toastchunkidtype = typid;
+ return typid;
+}
+
/*
* RelationGetIndexExpressions -- get the index expressions for an index
*
@@ -6530,6 +6566,7 @@ load_relcache_init_file(bool shared)
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
rel->rd_droppedSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_toastchunkidtype = InvalidOid;
rel->pgstat_info = NULL;
/*
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-10 17:06 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-10 17:06 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Thu, Sep 10, 2026 at 12:52 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Sep 10, 2026 at 08:18:57AM +0900, Michael Paquier wrote:
> > Hmm. Okay. I'll think about something here.
>
> Attached is a rebased v16 following cb298616463d. The patch set has
> been slightly re-organized, with the pieces that seem worth
> introducing first on top of the rest:
> - 0001 for binary upgrades.
0001 looks good to me. I reviewed v15 previously and there is no change in v16.
> - 0002 for the reloption toast_value_type
0002 looks good to me. The note in the docs about rewrites and pg_dump
and the tests are good additions. My previous comments on
parenthesizing the default arg and the typo in the docs are fixed.
> - 0003 for adding more Oid8 in the TOAST APIs compared to Oid.
0003 looks good to me. There is no change in this from the v15 patch,
and for the reorderbuffer.c change, I microbenchmarked uint32_hash and
tag_hash, and use of tag_hash seems fine to me.
> - 0004 for pg_column_toast_chunk_id() switched to Oid8
0004 looks good to me. I reviewed v15 previously and there is no change in v16.
> - 0005 for adding toast_value_type=oid8, with all the TOAST changes
> not related to the new vartag.
0005 looks good to me with the following comment. With this, one can
create a TOAST table with OID8 chunk_id but will only get OID4
behavior. Calling that out in the commit message seems fine to me, but
let us try to get this patch and 0007 (adding vartag_external) closer
or together. Also, the mention about the price one has to pay for
using OID8 (four extra bytes per out-of-line pointer) looks like it
addresses Greg's point on the max number of TOAST-able columns.
1/ An error would be better here when an unexpected TOAST typeid is
detected for some reason.
+ else
+ Assert(false);
> - 0006 is the battery of tests for all the cases found during
> development, grouped together: core things, amcheck, test_decoding..
0006 looks good to me. Tests cover most of the cases. Is it good to
cover the pg_dump and pg_restore case (OID4 to OID8 after changing the
reloption) as well, perhaps as a follow-up patch?
> - 0007 adds the new vartag_external.
>
> - 0008 is the tweak for toast_tuple_find_biggest_attribute(), which
> should perhaps be merged with 0007.
I will soon get to reviewing these two patches.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-10 21:14 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
1 sibling, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-10 21:14 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Thu, Sep 10, 2026 at 10:06 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> > - 0007 adds the new vartag_external.
> >
> > - 0008 is the tweak for toast_tuple_find_biggest_attribute(), which
> > should perhaps be merged with 0007.
>
> I will soon get to reviewing these two patches.
Here are some comments on 0007:
1/ Typo: "these can have different sizes."
+ * Note that TOAST pointers could have different vartags, for oid
+ * or oid8, and these can have a different sizes.
2/ Typo: "By default these values are chosen so that four chunk rows
will fit on a page..."?
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this
value is chosen
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
3/ Can these be Oid instead of uint32?
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
4/ Unlike ONDISK_OID which needs to be 18 to detect existing on-disk
pointers (like the comment on top of this structure definition
explains), having ONDISK_OID8 = 4 seems fine because there is no
backward compatibility requirement for OID8 chunk_ids yet. Is my
understanding correct here?
/*
* Type tag for the various sorts of "TOAST pointer" datums. The peculiar
* value for VARTAG_ONDISK_OID comes from a requirement for on-disk
* compatibility with a previous notion that the tag field was the pointer
* datum's length.
*/
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
5/ This looks good, but do we also need to have one for
varatt_external_oid? I don't think so, because it has been this way
for many years for varatt_external_oid (previously varatt_external).
* This struct must not contain any padding, because we sometimes compare
* these pointers using memcmp.
+StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==
+ sizeof(varatt_external_oid8),
+ "varatt_external_oid8 should have no padding");
6/ Missing typedefs.list entry for varatt_external_oid8.
7/ I think we can deduplicate most of the code to reduce the if (OID8)
else (OID) branching. I tried to do so and attached a diff on top of
v16-0007. Please have a look.
8/ Also, the code in toast_save_datum() now looks a bit complicated
and duplicated, and the comment about the race condition during
rewrite sits only in the OID8 block, which applies to both. I tried to
deduplicate it by moving the rewrite block to a separate function in
the attached diff. Please have a look.
Here are some comments on 0008:
1/ Caching the chunk_id type in the relcache entry seems fine, as it
avoids the syscache lookup costs in a relatively hot code path, for
every insert and update to a TOAST table.
2/ Also, upon ALTER TABLE setting the reloption, since the relcache
entry gets invalidated, we would get the updated reltoastrelid, right?
Say I change the reloption from OID to OID8, before and after the
chunk_id type would just be OID. Is my understanding correct?
3/ When RelationGetToastChunkIdType() returns InvalidOid, the caller
assumes the OID pointer size. I think that's fine, since a relation
without a TOAST table never externalizes anything, so the pointer size
is only the floor for inline-compression candidates and the OID vs
OID8 difference doesn't matter. It also matches today's behavior. Is
that the intent?
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
Attachments:
nocfbot-oid8-dedup.diff (application/octet-stream, 37.9K) download
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 86dc9cf51a9..69a22559661 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1681,6 +1681,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
uint32 va_extinfo;
CompactAttribute *thisatt;
vartag_external va_tag_value;
+ toast_external_data toast_ext_data;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1783,31 +1784,12 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
- /*
- * Must copy attr into toast_pointer for alignment considerations.
- * Branch on the tag to determine which pointer type to extract.
- */
- va_tag_value = VARTAG_EXTERNAL(attr);
- if (va_tag_value == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer8;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
- toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
- va_rawsize = toast_pointer8.va_rawsize;
- va_extinfo = toast_pointer8.va_extinfo;
- }
- else
- {
- varatt_external_oid toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
- va_rawsize = toast_pointer.va_rawsize;
- va_extinfo = toast_pointer.va_extinfo;
- }
+ /* Must copy attr into a decoded pointer for alignment considerations */
+ toast_external_info_get(attr, &toast_ext_data);
+ va_tag_value = toast_ext_data.tag;
+ toast_pointer_valueid = toast_ext_data.valueid;
+ va_rawsize = toast_ext_data.rawsize;
+ va_extinfo = toast_ext_data.extinfo;
/* Toasted attributes too large to be untoasted should never be stored */
if (va_rawsize > VARLENA_SIZE_LIMIT)
@@ -1817,13 +1799,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
va_rawsize,
VARLENA_SIZE_LIMIT));
- if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
+ cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1908,31 +1890,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;
toast_valueid = ta->va_valueid;
- extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
+ extsize = VARATT_EXTINFO_GET_EXTSIZE(ta->va_extinfo);
+ toast_typid = (ta->tag == VARTAG_ONDISK_OID8) ? OID8OID : OIDOID;
- /*
- * Setup a scan key to find chunks in toast table with matching va_valueid
- */
- if (ta->tag == VARTAG_ONDISK_OID8)
- {
- max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
-
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(toast_valueid));
- }
- else
- {
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum((Oid) toast_valueid));
- }
+ /* Set up a scan key to find chunks in the toast table by value ID */
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
+ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid,
+ toast_valueid);
last_chunk_seq = (extsize - 1) / max_chunk_size;
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 393135f1b9f..7e5863262b3 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,29 +225,15 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
+ toast_external_data toast_ext_data;
int32 extsize;
uint32 compress_method;
bool is_compressed;
- /* Extract extsize and compression info from the appropriate pointer */
- if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
- compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
- is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
- }
- else
- {
- varatt_external_oid toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
- compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
- is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
- }
+ toast_external_info_get(attr, &toast_ext_data);
+ extsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ compress_method = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
+ is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize);
/* fast path for non-compressed external datums */
if (!is_compressed)
@@ -359,6 +345,7 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
+ toast_external_data toast_ext_data;
int32 attrsize;
Oid toastrelid;
Oid8 valueid;
@@ -366,40 +353,17 @@ toast_fetch_datum(varlena *attr)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
- toastrelid = toast_pointer.va_toastrelid;
- valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ toast_external_info_get(attr, &toast_ext_data);
+ attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ toastrelid = toast_ext_data.toastrelid;
+ valueid = toast_ext_data.valueid;
- result = (varlena *) palloc(attrsize + VARHDRSZ);
+ result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
- else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
- }
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
else
- {
- varatt_external_oid toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
- toastrelid = toast_pointer.va_toastrelid;
- valueid = toast_pointer.va_valueid;
-
- result = (varlena *) palloc(attrsize + VARHDRSZ);
-
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
- else
- SET_VARSIZE(result, attrsize + VARHDRSZ);
- }
+ SET_VARSIZE(result, attrsize + VARHDRSZ);
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -437,6 +401,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
+ toast_external_data toast_ext_data;
int32 attrsize;
Oid toastrelid;
Oid8 valueid;
@@ -445,28 +410,11 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
- toastrelid = toast_pointer.va_toastrelid;
- valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);
- }
- else
- {
- varatt_external_oid toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
- toastrelid = toast_pointer.va_toastrelid;
- valueid = toast_pointer.va_valueid;
- is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);
- }
+ toast_external_info_get(attr, &toast_ext_data);
+ attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ toastrelid = toast_ext_data.toastrelid;
+ valueid = toast_ext_data.valueid;
+ is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize);
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
@@ -609,20 +557,10 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
- }
- else
- {
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
- }
+ toast_external_info_get(attr, &toast_ext_data);
+ result = toast_ext_data.rawsize;
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -679,20 +617,10 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);
- }
- else
- {
- varatt_external_oid toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
- }
+ toast_external_info_get(attr, &toast_ext_data);
+ result = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 200cee6be70..09cdf880e4f 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -260,23 +260,14 @@ toast_get_compression_id(varlena *attr)
* the external toast pointer. If compressed inline, fetch it from the
* toast compression header.
*/
- if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid8 toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
- if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))
- cmid = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);
- }
- else if (VARATT_IS_EXTERNAL_ONDISK_OID(attr))
- {
- varatt_external_oid toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- cmid = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
+ cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
}
else if (VARATT_IS_COMPRESSED(attr))
cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 84c4361af23..38a50883935 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -104,6 +104,55 @@ toast_compress_datum(Datum value, char cmethod)
}
}
+/* ----------
+ * toast_preserve_valueid -
+ *
+ * During a table rewrite that preserves rd_toastoid, we want to preserve
+ * toast value IDs too. If the datum previously had an external value from
+ * that same toast table, return its value ID so the caller can re-use it;
+ * otherwise return InvalidOid8. This works for both Oid and Oid8 value IDs,
+ * as the value ID is decoded independently of the pointer's vartag.
+ *
+ * There is a corner case here: the table rewrite might have to copy both
+ * live and recently-dead versions of a row, and those versions could easily
+ * reference the same toast value. When we copy the second or later version
+ * of such a row, preserving the value ID means we select one that's already
+ * in the new toast table. We detect that and set *data_todo to 0 so the
+ * caller falls through without writing the data again.
+ *
+ * While annoying and ugly-looking, this is a good thing because it ensures
+ * that we wind up with only one copy of the toast value when there is only
+ * one copy in the old toast table. Before we detected this case, we'd have
+ * made multiple copies, wasting space; and what's worse, the copies
+ * belonging to already-deleted heap tuples would not be reclaimed by VACUUM.
+ * ----------
+ */
+static Oid8
+toast_preserve_valueid(Relation toastrel, Oid toastoid,
+ varlena *oldexternal, int32 *data_todo)
+{
+ toast_external_data old;
+
+ if (!OidIsValid(toastoid) || oldexternal == NULL)
+ return InvalidOid8;
+
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ toast_external_info_get(oldexternal, &old);
+
+ /*
+ * Only re-use the value ID if the old pointer came from this same toast
+ * table. Since that is the very same relation, its chunk_id type
+ * necessarily matches, so no separate vartag check is needed.
+ */
+ if (old.toastrelid != toastoid)
+ return InvalidOid8;
+
+ if (toastrel_valueid_exists(toastrel, old.valueid))
+ *data_todo = 0;
+
+ return old.valueid;
+}
+
/* ----------
* toast_save_datum -
*
@@ -190,7 +239,7 @@ toast_save_datum(Relation rel, Datum value,
va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
/* Assert that the numbers look like it's compressed */
- Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
+ Assert(VARATT_EXTINFO_IS_COMPRESSED(va_extinfo, va_rawsize));
}
else
{
@@ -214,162 +263,45 @@ toast_save_datum(Relation rel, Datum value,
va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, and
- * determine the maximum chunk size based on the TOAST table's chunk_id
- * type.
- *
- * Normally we just choose an unused value within the toast table. But
- * during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value IDs too. So, if
- * rd_toastoid is set and we had a prior external value from that same
- * toast table, re-use its value ID. If we didn't have a prior external
- * value (which is a corner case, but possible if the table's attstorage
- * options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. For Oid8 tables,
- * value conflicts are not a concern.
+ * Choose the value ID for this toast value. During a table rewrite that
+ * preserves rd_toastoid we re-use the prior value ID if we can (see
+ * toast_preserve_valueid); otherwise we pick a fresh one. For Oid value
+ * IDs the fresh one must not conflict with old or new toast values; for
+ * Oid8 the ID space is large enough that conflicts are not a concern.
*/
- if (toast_typid == OID8OID)
+ if (toast_typid == OIDOID)
{
- if (!OidIsValid(rel->rd_toastoid))
- {
- /* normal case: just choose a new Oid8 */
- va_valueid = GetNewObjectId8();
- }
- else
+ va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid,
+ oldexternal, &data_todo);
+ if (va_valueid == InvalidOid)
{
- /* rewrite case: check to see if value was in old toast table */
- va_valueid = InvalidOid8;
- if (oldexternal != NULL)
- {
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
-
- /*
- * Only consider the old pointer if its vartag matches the
- * kind of pointer we are about to build.
- */
- if (VARATT_IS_EXTERNAL_ONDISK_OID8(oldexternal))
- {
- varatt_external_oid8 old_toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
- {
- /*
- * This value came from the old toast table; reuse its
- * ID.
- */
- va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);
-
- /*
- * There is a corner case here: the table rewrite
- * might have to copy both live and recently-dead
- * versions of a row, and those versions could easily
- * reference the same toast value. When we copy the
- * second or later version of such a row, reusing the
- * OID will mean we select an OID that's already in the
- * new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good
- * thing because it ensures that we wind up with only
- * one copy of the toast value when there is only one
- * copy in the old toast table. Before we detected
- * this case, we'd have made multiple copies, wasting
- * space; and what's worse, the copies belonging to
- * already-deleted heap tuples would not be reclaimed
- * by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel, va_valueid))
- {
- /*
- * Match, so short-circuit the data storage loop
- * below.
- */
- data_todo = 0;
- }
- }
- }
- }
- if (va_valueid == InvalidOid8)
- {
- /*
- * New value. For Oid8 we just pick a new ID without worrying
- * about conflicts.
- */
- va_valueid = GetNewObjectId8();
- }
+ /*
+ * Choose an unused OID. During a rewrite we must also avoid IDs
+ * that are still present in the old toast table.
+ */
+ do
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ while (OidIsValid(rel->rd_toastoid) &&
+ toastid_valueid_exists(rel->rd_toastoid, va_valueid));
}
-
- max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
- else
+ else if (toast_typid == OID8OID)
{
- if (!OidIsValid(rel->rd_toastoid))
- {
- /* normal case: just choose an unused OID */
- va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- }
- else
+ va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid,
+ oldexternal, &data_todo);
+ if (va_valueid == InvalidOid8)
{
- /* rewrite case: check to see if value was in old toast table */
- va_valueid = InvalidOid;
- if (oldexternal != NULL)
- {
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
-
- /* Only reuse a pointer of the matching type; same as above */
- if (VARATT_IS_EXTERNAL_ONDISK_OID(oldexternal))
- {
- varatt_external_oid old_toast_pointer;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
- {
- /*
- * This value came from the old toast table; reuse its
- * OID.
- */
- va_valueid = old_toast_pointer.va_valueid;
-
- /*
- * Corner case during table rewrite with multiple
- * versions of the same row. See above for details.
- */
- if (toastrel_valueid_exists(toastrel, va_valueid))
- {
- /*
- * Match, so short-circuit the data storage loop
- * below.
- */
- data_todo = 0;
- }
- }
- }
- }
- if (va_valueid == InvalidOid)
- {
- /*
- * new value; must choose a value that doesn't conflict in
- * either old or new toast table.
- */
- do
- {
- va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- va_valueid));
- }
+ /* The Oid8 space is large enough that we need not check conflicts */
+ va_valueid = GetNewObjectId8();
}
-
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
+ else
+ elog(ERROR, "unexpected TOAST value ID type %u", toast_typid);
+
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
/*
* Split up the item into chunks
@@ -486,6 +418,29 @@ toast_save_datum(Relation rel, Datum value,
return PointerGetDatum(result);
}
+/* ----------
+ * toast_valueid_scankey_init -
+ *
+ * Initialize a scan key that matches the value ID column of a TOAST table.
+ * The column type (OIDOID or OID8OID) selects the comparison operator, so
+ * callers need only supply the value ID widened to Oid8.
+ * ----------
+ */
+void
+toast_valueid_scankey_init(ScanKey entry, AttrNumber attno,
+ Oid toast_typid, Oid8 valueid)
+{
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ if (toast_typid == OID8OID)
+ ScanKeyInit(entry, attno,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ ScanKeyInit(entry, attno,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
+}
+
/* ----------
* toast_delete_datum -
*
@@ -496,6 +451,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
+ toast_external_data toast_ext_data;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -503,75 +459,30 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toastrelid;
- vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
/*
- * Determine the pointer type from the datum's vartag and extract the
- * toast relation OID and value ID accordingly. The vartag tells us
- * everything we need - no TOAST table schema lookup required.
+ * Decode the pointer to get the toast relation OID and value ID. The
+ * vartag tells us everything we need - no TOAST table schema lookup
+ * required.
*/
- tag = VARTAG_EXTERNAL(attr);
-
- if (tag == VARTAG_ONDISK_OID8)
- {
- varatt_external_oid8 toast_pointer8;
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);
- toastrelid = toast_pointer8.va_toastrelid;
-
- /*
- * Open the toast relation and its indexes
- */
- toastrel = table_open(toastrelid, RowExclusiveLock);
-
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
+ toast_external_info_get(attr, &toast_ext_data);
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));
- }
- else
- {
- varatt_external_oid toast_pointer;
-
- Assert(tag == VARTAG_ONDISK_OID);
-
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toastrelid = toast_pointer.va_toastrelid;
-
- /*
- * Open the toast relation and its indexes
- */
- toastrel = table_open(toastrelid, RowExclusiveLock);
+ /*
+ * Open the toast relation and its indexes
+ */
+ toastrel = table_open(toast_ext_data.toastrelid, RowExclusiveLock);
+ validIndex = toast_open_indexes(toastrel,
+ RowExclusiveLock,
+ &toastidxs,
+ &num_indexes);
- /* Fetch valid relation used for process */
- validIndex = toast_open_indexes(toastrel,
- RowExclusiveLock,
- &toastidxs,
- &num_indexes);
-
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
- }
+ /* Set up a scan key to find chunks with matching value ID */
+ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1,
+ TupleDescAttr(toastrel->rd_att, 0)->atttypid,
+ toast_ext_data.valueid);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -627,21 +538,9 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&num_indexes);
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
- if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(valueid));
- else
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum((Oid) valueid));
+ /* Set up a scan key to find chunks with matching value ID */
+ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid, valueid);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 31e6115de20..419adc1f3f7 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -650,10 +650,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- if (toast_typid == OID8OID)
- max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
- else
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
@@ -661,16 +658,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- if (toast_typid == OID8OID)
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(valueid));
- else
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum((Oid) valueid));
+ toast_valueid_scankey_init(&toastkey[0], (AttrNumber) 1, toast_typid, valueid);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 086ef31e4f2..69a479cd31d 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5175,15 +5175,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *varlena_pointer;
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
- varatt_external_oid8 toast_pointer8;
+ toast_external_data toast_ext_data;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
- int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5203,18 +5201,8 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer))
continue;
- if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))
- {
- VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);
- toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);
- rawsize = toast_pointer8.va_rawsize;
- }
- else
- {
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
- rawsize = toast_pointer.va_rawsize;
- }
+ toast_external_info_get(varlena_pointer, &toast_ext_data);
+ toast_valueid = toast_ext_data.valueid;
/*
* Check whether the toast tuple changed, replace if so.
@@ -5232,7 +5220,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(rawsize);
+ reconstructed = palloc0(toast_ext_data.rawsize);
ent->reconstructed = reconstructed;
@@ -5258,27 +5246,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- /* Verify size and set compression status based on pointer type */
- if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))
- {
- Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));
+ Assert(data_done == VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo));
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
- else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
- }
+ /* make sure its marked as compressed or not */
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
else
- {
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
-
- /* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
- else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
- }
+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0616db77555..f8c16bd4c87 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,6 +4261,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
+ toast_external_data toast_ext_data;
Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
@@ -4288,20 +4289,8 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))
- {
- varatt_external_oid8 toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
- }
- else
- {
- varatt_external_oid toast_pointer;
-
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_valueid;
- }
+ toast_external_info_get(attr, &toast_ext_data);
+ result = toast_ext_data.valueid;
PG_RETURN_OID8(result);
}
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index f065ea034e9..88301308544 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -12,6 +12,8 @@
#ifndef DETOAST_H
#define DETOAST_H
+#include "varatt.h"
+
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
* into a local "varatt_external_oid" toast pointer. This should be
@@ -36,6 +38,52 @@ do { \
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
+/*
+ * Decoded contents of an on-disk TOAST pointer, independent of whether the
+ * value ID is an Oid or an Oid8. This lets the many callers that only need
+ * the pointer's fields avoid open-coding the vartag branch. The value ID is
+ * always widened to Oid8; for a VARTAG_ONDISK_OID pointer it holds the 4-byte
+ * ID losslessly, and callers that need the narrow form cast back to Oid.
+ */
+typedef struct toast_external_data
+{
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ int32 rawsize; /* original data size (includes header) */
+ uint32 extinfo; /* saved size + compression method */
+ Oid8 valueid; /* value ID (widened from Oid if needed) */
+ Oid toastrelid; /* OID of the TOAST table containing it */
+} toast_external_data;
+
+/*
+ * Decode an on-disk TOAST pointer into a toast_external_data. The datum must
+ * be VARATT_IS_EXTERNAL_ONDISK.
+ */
+static inline void
+toast_external_info_get(const struct varlena *attr, toast_external_data *toast_ext_data)
+{
+ toast_ext_data->tag = VARTAG_EXTERNAL(attr);
+ if (toast_ext_data->tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_ext_data->rawsize = toast_pointer.va_rawsize;
+ toast_ext_data->extinfo = toast_pointer.va_extinfo;
+ toast_ext_data->valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ toast_ext_data->toastrelid = toast_pointer.va_toastrelid;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_ext_data->rawsize = toast_pointer.va_rawsize;
+ toast_ext_data->extinfo = toast_pointer.va_extinfo;
+ toast_ext_data->valueid = toast_pointer.va_valueid;
+ toast_ext_data->toastrelid = toast_pointer.va_toastrelid;
+ }
+}
+
/* ----------
* detoast_external_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index b7ef69a8f56..c58d0fa36a4 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -96,6 +96,13 @@
sizeof(int32) - \
VARHDRSZ)
+/*
+ * Select the chunk size for a TOAST table from its value ID column type
+ * (OIDOID or OID8OID).
+ */
+#define TOAST_MAX_CHUNK_SIZE(toast_typid) \
+ ((toast_typid) == OID8OID ? TOAST_OID8_MAX_CHUNK_SIZE : TOAST_OID_MAX_CHUNK_SIZE)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index bf45889a642..c0015f9cc6c 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -12,6 +12,7 @@
#ifndef TOAST_INTERNALS_H
#define TOAST_INTERNALS_H
+#include "access/skey.h"
#include "access/toast_compression.h"
#include "storage/lockdefs.h"
#include "utils/relcache.h"
@@ -52,6 +53,9 @@ extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
extern Datum toast_save_datum(Relation rel, Datum value,
varlena *oldexternal, uint32 options);
+extern void toast_valueid_scankey_init(ScanKey entry, AttrNumber attno,
+ Oid toast_typid, Oid8 valueid);
+
extern int toast_open_indexes(Relation toastrel,
LOCKMODE lock,
Relation **toastidxs,
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 1349a54371d..c8895df2651 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -564,33 +564,58 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
}
/*
- * Same for external Datums; but note argument is a struct
- * varatt_external_oid.
+ * The size and compression method of an externally-stored value are packed
+ * into a single 32-bit va_extinfo word: the low VARLENA_EXTSIZE_BITS hold the
+ * saved size, the top two bits the compression method. This encoding does not
+ * depend on the value ID width, so these helpers operate on the raw word and
+ * are shared by every on-disk TOAST pointer type.
*/
static inline Size
-VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_GET_EXTSIZE(uint32 extinfo)
{
- return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+ return extinfo & VARLENA_EXTSIZE_MASK;
}
static inline uint32
-VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_GET_COMPRESS_METHOD(uint32 extinfo)
{
- return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+ return extinfo >> VARLENA_EXTSIZE_BITS;
}
/*
- * Testing whether an externally-stored value is compressed now requires
- * comparing size stored in va_extinfo (the actual length of the external data)
+ * Testing whether an externally-stored value is compressed requires comparing
+ * the saved size stored in va_extinfo (the actual length of the external data)
* to rawsize (the original uncompressed datum's size). The latter includes
* VARHDRSZ overhead, the former doesn't. We never use compression unless it
* actually saves space, so we expect either equality or less-than.
*/
+static inline bool
+VARATT_EXTINFO_IS_COMPRESSED(uint32 extinfo, int32 rawsize)
+{
+ return VARATT_EXTINFO_GET_EXTSIZE(extinfo) < (Size) (rawsize - VARHDRSZ);
+}
+
+/*
+ * Same for external Datums; but note argument is a struct
+ * varatt_external_oid.
+ */
+static inline Size
+VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer)
+{
+ return VARATT_EXTINFO_GET_EXTSIZE(toast_pointer.va_extinfo);
+}
+
+static inline uint32
+VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
+{
+ return VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_pointer.va_extinfo);
+}
+
static inline bool
VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
{
- return VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer) <
- (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+ return VARATT_EXTINFO_IS_COMPRESSED(toast_pointer.va_extinfo,
+ toast_pointer.va_rawsize);
}
/*
@@ -600,20 +625,20 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
static inline Size
VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)
{
- return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+ return VARATT_EXTINFO_GET_EXTSIZE(toast_pointer.va_extinfo);
}
static inline uint32
VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)
{
- return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+ return VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_pointer.va_extinfo);
}
static inline bool
VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)
{
- return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <
- (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+ return VARATT_EXTINFO_IS_COMPRESSED(toast_pointer.va_extinfo,
+ toast_pointer.va_rawsize);
}
#endif
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7aedaafab90..dd5394ab3e0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4357,6 +4357,7 @@ timeout_params
timerCA
tlist_vinfo
toast_compress_header
+toast_external_data
tokenize_error_callback_arg
transferMode
transfer_thread_arg
@@ -4413,6 +4414,7 @@ vacuumingOptions
validate_string_relopt
varatt_expanded
varatt_external_oid
+varatt_external_oid8
varatt_indirect
varattrib_1b
varattrib_1b_e
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 03:42 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
1 sibling, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-11 03:42 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Sep 10, 2026 at 10:06:07AM -0700, Bharath Rupireddy wrote:
> On Thu, Sep 10, 2026 at 12:52 AM Michael Paquier <michael@paquier.xyz> wrote:
>> Attached is a rebased v16 following cb298616463d. The patch set has
>> been slightly re-organized, with the pieces that seem worth
>> introducing first on top of the rest:
>> - 0001 for binary upgrades.
>
> 0001 looks good to me. I reviewed v15 previously and there is no change in v16.
Applied that after a few tweaks, mostly some contents in pg_dump I
forgot to update.
>> - 0002 for the reloption toast_value_type
>
> 0002 looks good to me. The note in the docs about rewrites and pg_dump
> and the tests are good additions. My previous comments on
> parenthesizing the default arg and the typo in the docs are fixed.
Still missed an update of typedefs.list (not mandatory per-se, but I
tend to prefer clean indentation from the start).
>> - 0005 for adding toast_value_type=oid8, with all the TOAST changes
>> not related to the new vartag.
>
> 0005 looks good to me with the following comment. With this, one can
> create a TOAST table with OID8 chunk_id but will only get OID4
> behavior.
And that's exactly the point. In understand that I may be picky
regarding these splits, but I prefer this order rather than the
opposite of introducing the vartag and then the interfaces due to one
reason: untestable and not reachable dead code. Having 4-byte OID
pointers as a temporary measure for oid8 tables is not that bad in
comparison..
> Calling that out in the commit message seems fine to me, but
> let us try to get this patch and 0007 (adding vartag_external) closer
> or together. Also, the mention about the price one has to pay for
> using OID8 (four extra bytes per out-of-line pointer) looks like it
> addresses Greg's point on the max number of TOAST-able columns.
Will improve.
> 1/ An error would be better here when an unexpected TOAST typeid is
> detected for some reason.
> + else
> + Assert(false);
Yeah, I don't like much this pattern based on typid or even vartag
lookups. Will improve that a bit, probably without these asserts.
>> - 0006 is the battery of tests for all the cases found during
>> development, grouped together: core things, amcheck, test_decoding..
>
> 0006 looks good to me. Tests cover most of the cases. Is it good to
> cover the pg_dump and pg_restore case (OID4 to OID8 after changing the
> reloption) as well, perhaps as a follow-up patch?
Yes, I guess I should. One thing that slightly annoys me is that we
don't have zero dump/restore tests for TOAST tables in binary upgrade
mode, and no checks related to their relfilenode and with their SQL
functions produces.. That's not the problem of this patch set, but
let's close the gap anyway as an initial, separate, improvement.
>> - 0007 adds the new vartag_external.
>>
>> - 0008 is the tweak for toast_tuple_find_biggest_attribute(), which
>> should perhaps be merged with 0007.
>
> I will soon get to reviewing these two patches.
Thanks again for the reviews. I still need to follow with your second
mail, which will include a rebase and all the comments of this message
addressed.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 07:33 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 3 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-11 07:33 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Sep 10, 2026 at 02:14:00PM -0700, Bharath Rupireddy wrote:
> 1/ Typo: "these can have different sizes."
> 2/ Typo: "By default these values are chosen so that four chunk rows
> will fit on a page..."?
> 6/ Missing typedefs.list entry for varatt_external_oid8.
Fixed.
> 3/ Can these be Oid instead of uint32?
>
> + uint32 va_valueid_lo; /* Low 32 bits of value ID */
> + uint32 va_valueid_hi; /* High 32 bits of value ID */
An Oid8 is not a subset of two Oids, so nope.
> 4/ Unlike ONDISK_OID which needs to be 18 to detect existing on-disk
> pointers (like the comment on top of this structure definition
> explains), having ONDISK_OID8 = 4 seems fine because there is no
> backward compatibility requirement for OID8 chunk_ids yet. Is my
> understanding correct here?
18 is just used as an historical artifact. Using 4 is fine, yes.
> 5/ This looks good, but do we also need to have one for
> varatt_external_oid? I don't think so, because it has been this way
> for many years for varatt_external_oid (previously varatt_external).
Hmm. Why not. Added one with 1497ea77edd2.
> 7/ I think we can deduplicate most of the code to reduce the if (OID8)
> else (OID) branching. I tried to do so and attached a diff on top of
> v16-0007. Please have a look.
> 8/ Also, the code in toast_save_datum() now looks a bit complicated
> and duplicated, and the comment about the race condition during
> rewrite sits only in the OID8 block, which applies to both. I tried to
> deduplicate it by moving the rewrite block to a separate function in
> the attached diff. Please have a look.
OK, that's a big chunk that reduces by close to 30% my previous bigger
chunk of code. After a close read, this changes three things:
- Introduction of toast_valueid_scankey_init(). It would be weird to
add that in the last patch while this fixes some of the bloat from
"Add support for oid8 TOAST values" for the scan keys. Find that
pretty nice.
- toast_external_data and toast_external_info_get() is what reduces a
lot of the chunk of data. Hannu had an equivalent on his other thread
for the tids where he wanted to extract all the fields individually
into an intermediate structure before passing it around, instead of
having a bunch of if/else block that depend on the vartag_external.
That's actually a super nice addition.
- toast_preserve_valueid() is also something that I've found rather
nice. The inner loop handling the OID[8] assignment becomes super
minimal, with the rewrite reasoning in the new routine.
The last two points make most sense in the last patch, because they
depend on the new vartag from what I can see.
Finally, some numbers for the last patch that introduces the new
vartag (with the biggest attribute fix included in last patch):
- v17:
16 files changed, 430 insertions(+), 212 deletions(-)
- v16:
15 files changed, 661 insertions(+), 219 deletions(-)
In short you are cutting 240 lines of code for the last changes with
your suggestions, and make the code much more readable. I'd say that
this is nice.
> 2/ Also, upon ALTER TABLE setting the reloption, since the relcache
> entry gets invalidated, we would get the updated reltoastrelid, right?
> Say I change the reloption from OID to OID8, before and after the
> chunk_id type would just be OID. Is my understanding correct?
Yes. That should be what happens.
> 3/ When RelationGetToastChunkIdType() returns InvalidOid, the caller
> assumes the OID pointer size. I think that's fine, since a relation
> without a TOAST table never externalizes anything, so the pointer size
> is only the floor for inline-compression candidates and the OID vs
> OID8 difference doesn't matter. It also matches today's behavior. Is
> that the intent?
Yes, using the OID size is for HEAD, mostly. I did not want to break
that, and an oid8-based size would be an outlier, so an oid-based size
as fallback is the natural thing to do, IMO.
Attached is a rebased v17, with the last 5 patches and your
refactorings integrated in a cleaner manner, based on my points from
above.
Thanks again for all the reviews.
--
Michael
Attachments: download all (6 files, .tar.gz)
v17-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch (text/plain, 16.1K) download
From aa4b7411475f4ff7c745f74836c9853c70623087 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 13:40:41 +0900
Subject: [PATCH v17 1/5] Refactor some TOAST value ID code to use Oid8 instead
of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 23 +++++---
contrib/amcheck/verify_heapam.c | 55 +++++++++++--------
6 files changed, 60 insertions(+), 44 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index b2346fd75bd4..aaae6e0fef69 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index ff03a2b816fc..ebec31a22e40 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1987,7 +1987,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 0f234661bfa9..ca5ed8dfabca 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);
-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)
+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)
+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 3b7c658a90e6..81154c17376c 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -729,7 +729,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",
+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -741,13 +741,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",
+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -756,7 +756,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",
+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -785,7 +785,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",
+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 457d544eec93..dd4b6e8f7f45 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */
+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -5010,7 +5010,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);
+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5034,8 +5034,10 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;
+ Oid8 chunk_id;
int32 chunk_seq;
+ Oid valueid_type;
+ Datum valueid_datum;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
@@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));
+ valueid_type = TupleDescAttr(desc, 0)->atttypid;
+ valueid_datum = fastgetattr(newtup, 1, desc, &isnull);
+ if (valueid_type == OID8OID)
+ chunk_id = DatumGetObjectId8(valueid_datum);
+ else
+ chunk_id = DatumGetObjectId(valueid_datum);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
@@ -5061,11 +5068,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",
+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5174,6 +5181,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5194,13 +5202,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,
+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index bbaaec89f8ae..33b00fa4dc5c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1564,9 +1564,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 chunksize;
int32 expected_size;
int32 max_chunk_size;
+ Oid8 toast_valueid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1574,16 +1576,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",
+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",
+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1594,8 +1596,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",
+ toast_valueid,
chunk_seq));
return;
}
@@ -1614,8 +1616,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",
+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1626,8 +1628,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",
+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1637,8 +1639,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",
+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1669,6 +1671,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external_oid toast_pointer;
@@ -1777,12 +1780,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",
- toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1809,16 +1813,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",
- toast_pointer.va_valueid, cmid));
+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",
+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",
+ toast_pointer_valueid));
return true;
}
@@ -1826,8 +1830,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",
- toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",
+ toast_pointer_valueid));
return true;
}
@@ -1873,6 +1877,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1903,14 +1908,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;
+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",
- ta->toast_pointer.va_valueid));
+ psprintf("toast value " OID8_FORMAT " not found in toast table",
+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",
- ta->toast_pointer.va_valueid,
+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",
+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.55.0
v17-0002-Switch-pg_column_toast_chunk_id-return-value-fro.patch (text/plain, 5.7K) download
From 5466a0440149f35b48fe4c39b8bd49a9283e07de Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v17 2/5] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 74d386b868bf..f46427258e3d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7907,7 +7907,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',
+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index f6a41e709ae0..4533dac11a6d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);
+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out
index cb14ddcee34b..21859fc0dce2 100644
--- a/src/test/isolation/expected/cluster-toast-value-reuse.out
+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out
@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec
index 9a2d10600b39..96cf899db296 100644
--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec
+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec
@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);
+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
index fa76bcf28ef9..cc8f034d0167 100644
--- a/src/test/modules/injection_points/specs/repack_toast.spec
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);
+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 2990e0c4f286..35d6ee2ce9eb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 950d9ab1a4a1..94b91964c046 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)
+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 54eeb42e5bce..70d762e73328 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>
+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.55.0
v17-0003-Add-support-for-oid8-TOAST-values.patch (text/plain, 20.1K) download
From 21d7e746779ae13008becc83e204b56a0d4b2860 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 11 Sep 2026 16:17:53 +0900
Subject: [PATCH v17 3/5] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, unfortunately casted to OID for now, as we do not have
a vartag_external able to store Oid8 values yet.
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/access/toast_internals.h | 4 +
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 3 +-
src/backend/access/common/toast_internals.c | 91 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 7 +-
src/backend/catalog/toasting.c | 25 +++++-
src/test/regress/expected/reloptions.out | 15 +++-
src/test/regress/sql/reloptions.sql | 8 ++
doc/src/sgml/ref/create_table.sgml | 5 +-
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 13 +--
12 files changed, 137 insertions(+), 45 deletions(-)
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index bf45889a6428..c0015f9cc6cf 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -12,6 +12,7 @@
#ifndef TOAST_INTERNALS_H
#define TOAST_INTERNALS_H
+#include "access/skey.h"
#include "access/toast_compression.h"
#include "storage/lockdefs.h"
#include "utils/relcache.h"
@@ -52,6 +53,9 @@ extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
extern Datum toast_save_datum(Relation rel, Datum value,
varlena *oldexternal, uint32 options);
+extern void toast_valueid_scankey_init(ScanKey entry, AttrNumber attno,
+ Oid toast_typid, Oid8 valueid);
+
extern int toast_open_indexes(Relation toastrel,
LOCKMODE lock,
Relation **toastidxs,
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index a88549dcc555..d61432de705e 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 82491a31a9a4..ea9a04179090 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
@@ -578,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")
+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..f17e7b8813cb 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,6 +132,7 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
@@ -200,24 +202,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
}
else
{
@@ -263,17 +273,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
}
}
@@ -303,7 +318,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -366,6 +384,27 @@ toast_save_datum(Relation rel, Datum value,
return PointerGetDatum(result);
}
+/* ----------
+ * toast_valueid_scankey_init -
+ *
+ * Initialize a scan key that matches the value ID column of a TOAST table.
+ * ----------
+ */
+void
+toast_valueid_scankey_init(ScanKey entry, AttrNumber attno,
+ Oid toast_typid, Oid8 valueid)
+{
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ if (toast_typid == OID8OID)
+ ScanKeyInit(entry, attno,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ ScanKeyInit(entry, attno,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
+}
+
/* ----------
* toast_delete_datum -
*
@@ -405,10 +444,9 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1,
+ TupleDescAttr(toastrel->rd_att, 0)->atttypid,
+ toast_pointer.va_valueid);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +493,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +501,13 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid,
+ valueid);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..b9c89593ecf4 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +657,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ toast_valueid_scankey_init(&toastkey[0], (AttrNumber) 1, toast_typid, valueid);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index a4a7b7c91b31..a7079661666c 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -173,6 +174,9 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
case STDRD_OPTION_TOAST_VALUE_TYPE_OID:
toast_chunkid_typid = OIDOID;
break;
+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID8:
+ toast_chunkid_typid = OID8OID;
+ break;
case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID:
elog(ERROR, "unexpected toast_value_type value %d",
value_type);
@@ -210,7 +214,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -235,6 +240,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -353,7 +371,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
+ else
+ opclassIds[0] = OID_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out
index 9b4c8587aa49..95e4528f7793 100644
--- a/src/test/regress/expected/reloptions.out
+++ b/src/test/regress/expected/reloptions.out
@@ -199,7 +199,7 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
ERROR: unrecognized parameter "toast_value_type"
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
ERROR: invalid value for enum option "toast_value_type": int8
-DETAIL: Valid values are "oid".
+DETAIL: Valid values are "oid" and "oid8".
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
reloptions
@@ -207,6 +207,19 @@ SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
{toast_value_type=oid}
(1 row)
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ chunk_id_type
+---------------
+ oid
+(1 row)
+
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
DROP TABLE reloptions_test2;
-- relkinds not supported.
CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql
index 503d15d16e51..80939ee8b5bd 100644
--- a/src/test/regress/sql/reloptions.sql
+++ b/src/test/regress/sql/reloptions.sql
@@ -117,6 +117,14 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
DROP TABLE reloptions_test2;
-- relkinds not supported.
CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 79622c1a23d6..6f9a87ef449f 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1663,7 +1663,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<para>
Specifies the attribute type of <literal>chunk_id</literal> to use when
creating a <acronym>TOAST</acronym> relation for this table. The
- default is <literal>oid</literal>.
+ default is <literal>oid</literal>; <literal>oid8</literal> assigns
+ <type>oid8</type> instead, which allows a larger number of distinct
+ <acronym>TOAST</acronym>ed values at the price of four extra bytes per
+ out-of-line pointer.
This parameter cannot be set for <acronym>TOAST</acronym> tables.
</para>
<para>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 90aae3defcbc..21bd67c11ea0 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an OID8 identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 33b00fa4dc5c..c89fab8764ac 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -28,7 +28,6 @@
#include "storage/procarray.h"
#include "storage/read_stream.h"
#include "utils/builtins.h"
-#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/tuplestore.h"
@@ -1876,8 +1875,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1885,10 +1888,8 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid,
+ ta->toast_pointer.va_valueid);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v17-0004-Add-battery-of-tests-related-oid8.patch (text/plain, 37.4K) download
From d32734c973ea57ead8afce118e30e92cbd910ffb Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 10 Sep 2026 16:24:52 +0900
Subject: [PATCH v17 4/5] Add battery of tests related oid8
The tests added by this commit cover a large ground in terms of the
introduction of oid8 as external TOAST pointer, with the following areas
covered, as found during development of the feature:
- Tables with oid8 TOAST tables in general, under strings.sql.
- Relation rewrites do not affect the TOAST relation after initial
creation.
- oid8 TOAST with pg_column_compression().
- test_decoding, for external and extended TOAST data with oid8
pointers.
- amcheck, with oid8 TOAST relations, external and extended formats with
heap verified.
- UPDATE with out-of-line datum that belongs to another TOAST table.
This tracks a bug in toast_tuple_init(), where external vartags could
be mixed once multiple types are introduced. That's something that
absan would trigger.
These tests are added before the commit that has adds the new external
vartag, just because it is what it is.
---
src/test/regress/expected/compression.out | 41 +++
src/test/regress/expected/strings.out | 300 +++++++++++++++++++---
src/test/regress/sql/compression.sql | 22 ++
src/test/regress/sql/strings.sql | 181 ++++++++++---
contrib/amcheck/expected/check_heap.out | 40 +++
contrib/amcheck/sql/check_heap.sql | 27 ++
contrib/test_decoding/expected/toast.out | 39 +++
contrib/test_decoding/sql/toast.sql | 20 ++
8 files changed, 595 insertions(+), 75 deletions(-)
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 09f198149aa4..e9128193515a 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
(1 row)
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+ uncompressed
+--------------
+ t
+(1 row)
+
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+ uncompressed
+--------------
+ t
+(1 row)
+
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+DROP TABLE toastcomp_oid, toastcomp_oid8;
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 313c50394650..7e3a641259c2 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2105,7 +2198,140 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index ce5ea37a660c..229143f06ddd 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2;
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+DROP TABLE toastcomp_oid, toastcomp_oid8;
+
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 38946e8954df..4d6a1e02359c 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,194 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index 979e5e84e723..8686fba712d0 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -199,6 +199,44 @@ SELECT * FROM verify_heapam('test_partition',
-------+--------+--------+-----
(0 rows)
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+ relname | chunk_id_type
+-----------------+---------------
+ test_toast_oid | oid
+ test_toast_oid8 | oid8
+(2 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -232,6 +270,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 1745bae634e5..eb46fe710351 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -112,6 +112,31 @@ SELECT * FROM verify_heapam('test_partition',
startblock := NULL,
endblock := NULL);
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -139,6 +164,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
endblock := NULL);
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.out
index a757e7dc8d54..8bef6cc32af4 100644
--- a/contrib/test_decoding/expected/toast.out
+++ b/contrib/test_decoding/expected/toast.out
@@ -382,6 +382,45 @@ SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot',
COMMIT
(4 rows)
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ shortened | len
+------------------------------------------------------------------------------------+--------
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:1 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:2 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: old-key: id[integer]:1 ne..nchanged-toast-datum | 116
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: id[integer]:2 data[text]:..bcdefghijabcdefghij' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: DELETE: id[integer]:11 | 49
+ table public.toasted_oid8: DELETE: id[integer]:2 | 48
+ COMMIT | 6
+(16 rows)
+
+DROP TABLE toasted_oid8;
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sql
index d1c560a174d6..8a2c49a5c9a7 100644
--- a/contrib/test_decoding/sql/toast.sql
+++ b/contrib/test_decoding/sql/toast.sql
@@ -324,4 +324,24 @@ INSERT INTO tbl2 VALUES(1);
commit;
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+DROP TABLE toasted_oid8;
+
SELECT pg_drop_replication_slot('regression_slot');
--
2.55.0
v17-0005-Add-support-for-TOAST-pointers-as-oid8.patch (text/plain, 46.8K) download
From 5199347bfc9a3daa45a2058c721167b5d0b1ca1a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 11 Sep 2026 15:57:31 +0900
Subject: [PATCH v17 5/5] Add support for TOAST pointers as oid8
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 49 ++++
src/include/access/heaptoast.h | 22 +-
src/include/utils/rel.h | 9 +
src/include/utils/relcache.h | 1 +
src/include/varatt.h | 78 ++++--
src/backend/access/common/detoast.c | 75 +++---
src/backend/access/common/toast_compression.c | 8 +-
src/backend/access/common/toast_internals.c | 245 ++++++++++--------
src/backend/access/heap/heaptoast.c | 2 +-
src/backend/access/table/toast_helper.c | 22 +-
.../replication/logical/reorderbuffer.c | 15 +-
src/backend/utils/adt/varlena.c | 8 +-
src/backend/utils/cache/relcache.c | 37 +++
doc/src/sgml/storage.sgml | 12 +-
contrib/amcheck/verify_heapam.c | 57 ++--
src/tools/pgindent/typedefs.list | 2 +
16 files changed, 430 insertions(+), 212 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..e9d8076f77cd 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -12,6 +12,8 @@
#ifndef DETOAST_H
#define DETOAST_H
+#include "varatt.h"
+
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
* into a local "varatt_external_oid" toast pointer. This should be
@@ -30,9 +32,56 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
+/*
+ * Decoded contents of an on-disk TOAST pointer, independent of whether the
+ * value ID type.
+ */
+typedef struct toast_external_data
+{
+ vartag_external tag; /* VARTAG_ONDISK_* */
+ int32 rawsize; /* original data size (includes header) */
+ uint32 extinfo; /* saved size + compression method */
+ Oid8 valueid; /* value ID (can be widened from Oid) */
+ Oid toastrelid; /* OID of the TOAST table containing it */
+} toast_external_data;
+
+/*
+ * Decode an on-disk TOAST pointer into a toast_external_data.
+ */
+static inline void
+toast_external_info_get(const struct varlena *attr, toast_external_data *toast_ext_data)
+{
+ Assert(VARATT_IS_EXTERNAL_ONDISK(attr));
+
+ toast_ext_data->tag = VARTAG_EXTERNAL(attr);
+ if (toast_ext_data->tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_ext_data->rawsize = toast_pointer.va_rawsize;
+ toast_ext_data->extinfo = toast_pointer.va_extinfo;
+ toast_ext_data->valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ toast_ext_data->toastrelid = toast_pointer.va_toastrelid;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_ext_data->rawsize = toast_pointer.va_rawsize;
+ toast_ext_data->extinfo = toast_pointer.va_extinfo;
+ toast_ext_data->valueid = toast_pointer.va_valueid;
+ toast_ext_data->toastrelid = toast_pointer.va_toastrelid;
+ }
+}
+
/* ----------
* detoast_external_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..4c3fda8a5f9a 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,19 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
+/*
+ * Select the chunk size for a TOAST table from its value type.
+ */
+#define TOAST_MAX_CHUNK_SIZE(toast_typid) \
+ ((toast_typid) == OID8OID ? TOAST_OID8_MAX_CHUNK_SIZE : TOAST_OID_MAX_CHUNK_SIZE)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index d61432de705e..916b67f5504f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -250,6 +250,15 @@ typedef struct RelationData
*/
Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */
+ /*
+ * Type OID of the "chunk_id" column of this relation's TOAST table, i.e.
+ * OIDOID or OID8OID.
+ *
+ * This data is filled on demand by RelationGetToastChunkIdType(), not
+ * at relcache build time, so as to save in syscache lookups.
+ */
+ Oid rd_toastchunkidtype;
+
bool pgstat_enabled; /* should relation stats be counted */
/* use "struct" here to avoid needing to include pgstat.h: */
struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 89c27aa1529f..e17c84908332 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok);
extern Oid RelationGetReplicaIndex(Relation relation);
+extern Oid RelationGetToastChunkIdType(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
extern List *RelationGetDummyIndexExpressions(Relation relation);
extern List *RelationGetIndexPredicate(Relation relation);
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 89b47db08332..77041f7673a9 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -42,6 +42,44 @@ StaticAssertDecl((sizeof(int32) + sizeof(uint32) + 2 * sizeof(Oid)) ==
sizeof(varatt_external_oid),
"varatt_external_oid should have no padding");
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except for the value ID, which is 8 bytes wide. The value ID is split
+ * into two uint32 to force alignment.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==
+ sizeof(varatt_external_oid8),
+ "varatt_external_oid8 should have no padding");
+
+/* Get/set the 64-bit value ID from the lo/hi halves */
+static inline Oid8
+VARATT_EXTERNAL_OID8_GET_VALUEID(varatt_external_oid8 toast_pointer)
+{
+ return ((Oid8) toast_pointer.va_valueid_lo) |
+ (((Oid8) toast_pointer.va_valueid_hi) << 32);
+}
+
+static inline void
+VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id)
+{
+ toast_pointer->va_valueid_lo = (uint32) id;
+ toast_pointer->va_valueid_hi = (uint32) (id >> 32);
+}
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -91,6 +129,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -112,6 +151,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -365,7 +406,12 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
}
/* Is varlena datum an indirect pointer? */
@@ -508,43 +554,31 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
}
/*
- * Same for external Datums; but note argument is a struct
- * varatt_external_oid.
+ * Same for external Datums, saved into an va_extinfo.
*/
static inline Size
-VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_GET_EXTSIZE(uint32 extinfo)
{
- return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+ return extinfo & VARLENA_EXTSIZE_MASK;
}
static inline uint32
-VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_GET_COMPRESS_METHOD(uint32 extinfo)
{
- return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+ return extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
- * Testing whether an externally-stored value is compressed now requires
- * comparing size stored in va_extinfo (the actual length of the external data)
+ * Testing whether an externally-stored value is compressed requires comparing
+ * the saved size stored in va_extinfo (the actual length of the external data)
* to rawsize (the original uncompressed datum's size). The latter includes
* VARHDRSZ overhead, the former doesn't. We never use compression unless it
* actually saves space, so we expect either equality or less-than.
*/
static inline bool
-VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_IS_COMPRESSED(uint32 extinfo, int32 rawsize)
{
- return VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer) <
- (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+ return VARATT_EXTINFO_GET_EXTSIZE(extinfo) < (Size) (rawsize - VARHDRSZ);
}
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..7e5863262b3d 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,12 +225,18 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
+ extsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ compress_method = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
+ is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize);
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +246,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,14 +257,9 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,20 +345,22 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toast_external_info_get(attr, &toast_ext_data);
+ attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ toastrelid = toast_ext_data.toastrelid;
+ valueid = toast_ext_data.valueid;
result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
else
SET_VARSIZE(result, attrsize + VARHDRSZ);
@@ -369,10 +372,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +401,27 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
+ attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ toastrelid = toast_ext_data.toastrelid;
+ valueid = toast_ext_data.valueid;
+ is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize);
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +434,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +447,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +456,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +557,10 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ toast_external_info_get(attr, &toast_ext_data);
+ result = toast_ext_data.rawsize;
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +617,10 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toast_external_info_get(attr, &toast_ext_data);
+ result = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 3b1bd6519261..09cdf880e4ff 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -262,12 +262,12 @@ toast_get_compression_id(varlena *attr)
*/
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- cmid = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
+ cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
}
else if (VARATT_IS_COMPRESSED(attr))
cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index f17e7b8813cb..328f98e338b9 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,7 +25,6 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -104,6 +103,57 @@ toast_compress_datum(Datum value, char cmethod)
}
}
+/* ----------
+ * toast_preserve_valueid -
+ *
+ * During a table rewrite that preserves rd_toastoid, we want to preserve
+ * toast value IDs too. If the datum previously had an external value from
+ * that same toast table, return its value ID so the caller can re-use it,
+ * otherwise return InvalidOid8.
+ *
+ * This works for both Oid and Oid8 value IDs, as the value ID is decoded
+ * independently of the pointer's vartag.
+ *
+ * There is a corner case here: the table rewrite might have to copy both
+ * live and recently-dead versions of a row, and those versions could easily
+ * reference the same toast value. When we copy the second or later version
+ * of such a row, preserving the value ID means we select one that's already
+ * in the new toast table. We detect that and set *data_todo to 0 so the
+ * caller falls through without writing the data again.
+ *
+ * While annoying and ugly-looking, this is a good thing because it ensures
+ * that we wind up with only one copy of the toast value when there is only
+ * one copy in the old toast table. Before we detected this case, we'd have
+ * made multiple copies, wasting space; and what's worse, the copies
+ * belonging to already-deleted heap tuples would not be reclaimed by VACUUM.
+ * ----------
+ */
+static Oid8
+toast_preserve_valueid(Relation toastrel, Oid toastoid,
+ varlena *oldexternal, int32 *data_todo)
+{
+ toast_external_data old;
+
+ if (!OidIsValid(toastoid) || oldexternal == NULL)
+ return InvalidOid8;
+
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ toast_external_info_get(oldexternal, &old);
+
+ /*
+ * Only re-use the value ID if the old pointer came from this same toast
+ * table. Since that is the very same relation, its chunk_id type
+ * necessarily matches, so no separate vartag check is needed.
+ */
+ if (old.toastrelid != toastoid)
+ return InvalidOid8;
+
+ if (toastrel_valueid_exists(toastrel, old.valueid))
+ *data_todo = 0;
+
+ return old.valueid;
+}
+
/* ----------
* toast_save_datum -
*
@@ -125,14 +175,20 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
- Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ Oid toast_typid = RelationGetToastChunkIdType(rel);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
@@ -164,28 +220,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert(VARATT_EXTINFO_IS_COMPRESSED(va_extinfo, va_rawsize));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -197,101 +257,49 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or OID8 TOAST relations.
- *
- * Normally we just choose an unused value within the toast table. But
- * during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value IDs too. So, if
- * rd_toastoid is set and we had a prior external value from that same
- * toast table, re-use its value ID. If we didn't have a prior external
- * value (which is a corner case, but possible if the table's attstorage
- * options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * Choose the value ID for this toast value. During a table rewrite that
+ * preserves rd_toastoid we re-use the prior value ID if we can (see
+ * toast_preserve_valueid); otherwise we pick a fresh one. For Oid value
+ * IDs the fresh one must not conflict with old or new toast values; for
+ * Oid8 the ID space is large enough that conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
- else
+ va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid,
+ oldexternal, &data_todo);
+ if (va_valueid == InvalidOid8)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ /* The Oid8 space is large enough that we need not check conflicts */
+ va_valueid = GetNewObjectId8();
}
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
- {
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
- {
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
-
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
- {
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
- }
- }
- }
- if (toast_pointer.va_valueid == InvalidOid)
+ va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid,
+ oldexternal, &data_todo);
+ if (va_valueid == InvalidOid)
{
/*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
+ * Choose an unused OID. During a rewrite we must also avoid IDs
+ * that are still present in the old toast table.
*/
- if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
- else
- {
- do
- {
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
- }
+ do
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ while (OidIsValid(rel->rd_toastoid) &&
+ toastid_valueid_exists(rel->rd_toastoid, va_valueid));
}
}
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
+
/*
* Split up the item into chunks
*/
@@ -313,15 +321,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ t_values[0] = ObjectId8GetDatum(va_valueid);
else
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -375,11 +383,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(&toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -415,7 +446,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -427,26 +458,26 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /*
+ * Decode the pointer to get the toast relation OID and value ID. The
+ * vartag tells us everything we need - no TOAST table schema lookup
+ * required.
+ */
+ toast_external_info_get(attr, &toast_ext_data);
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
-
- /* Fetch valid relation used for process */
+ toastrel = table_open(toast_ext_data.toastrelid, RowExclusiveLock);
validIndex = toast_open_indexes(toastrel,
RowExclusiveLock,
&toastidxs,
&num_indexes);
- /*
- * Setup a scan key to find chunks with matching va_valueid
- */
+ /* Set up a scan key to find chunks with matching value ID */
toast_valueid_scankey_init(&toastkey, (AttrNumber) 1,
TupleDescAttr(toastrel->rd_att, 0)->atttypid,
- toast_pointer.va_valueid);
+ toast_ext_data.valueid);
/*
* Find all the chunks. (We don't actually care whether we see them in
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index b9c89593ecf4..a3d5cdecb3c1 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -649,7 +649,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&num_indexes);
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6f2d691cc681..2613d9dc095b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc)
/*
* If the old value is stored on disk, check if it has changed so
* we have to delete it later.
+ *
+ * Note that TOAST pointers could have different vartags, for oid
+ * or oid8, and these can have different sizes.
*/
if (att->attlen == -1 && !ttc->ttc_oldisnull[i] &&
VARATT_IS_EXTERNAL_ONDISK(old_value))
{
if (ttc->ttc_isnull[i] ||
!VARATT_IS_EXTERNAL_ONDISK(new_value) ||
+ VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) ||
memcmp(old_value, new_value,
VARSIZE_EXTERNAL(old_value)) != 0)
{
@@ -171,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
- * if not, no benefit is to be expected by compressing it.
+ * The column must be larger than the TOAST pointer that would replace it;
+ * if not, no benefit is to be expected by compressing it. Note that this
+ * choice depends on the TOAST value type, oid or oid8.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -184,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+ int32 biggest_size;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /*
+ * Size of the TOAST pointer this relation would use. A relation without
+ * a TOAST table cannot have any of its attributes moved out-of-line, but
+ * it can still have some of them compressed. Fall back to the oid size
+ * in that case.
+ */
+ if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID)
+ biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE);
+ else
+ biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dd4b6e8f7f45..69a479cd31da 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5175,7 +5175,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *varlena_pointer;
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
@@ -5198,11 +5198,11 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena_pointer = (varlena *) DatumGetPointer(attrs[natt]);
/* no need to do anything if the tuple isn't external */
- if (!VARATT_IS_EXTERNAL(varlena_pointer))
+ if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ toast_external_info_get(varlena_pointer, &toast_ext_data);
+ toast_valueid = toast_ext_data.valueid;
/*
* Check whether the toast tuple changed, replace if so.
@@ -5220,7 +5220,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(toast_ext_data.rawsize);
ent->reconstructed = reconstructed;
@@ -5245,10 +5245,11 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ Assert(data_done == VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo));
/* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
else
SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 4533dac11a6d..f8c16bd4c871 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,7 +4261,8 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
+ Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4288,9 +4289,10 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
+ result = toast_ext_data.valueid;
- PG_RETURN_OID8(toast_pointer.va_valueid);
+ PG_RETURN_OID8(result);
}
/*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index f475d7039772..65dfdbda100a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1205,6 +1205,9 @@ retry:
relation->rd_fkeylist = NIL;
relation->rd_fkeyvalid = false;
+ /* TOAST type data is not loaded till asked for */
+ relation->rd_toastchunkidtype = InvalidOid;
+
/* partitioning data is not loaded till asked for */
relation->rd_partkey = NULL;
relation->rd_partkeycxt = NULL;
@@ -5093,6 +5096,39 @@ RelationGetReplicaIndex(Relation relation)
return relation->rd_replidindex;
}
+/*
+ * RelationGetToastChunkIdType -- get the type of the relation's TOAST
+ * table "chunk_id" column
+ *
+ * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST
+ * table.
+ */
+Oid
+RelationGetToastChunkIdType(Relation relation)
+{
+ Oid toastrelid = relation->rd_rel->reltoastrelid;
+ Oid typid;
+
+ /* Quick exit if we already computed the value */
+ if (OidIsValid(relation->rd_toastchunkidtype))
+ return relation->rd_toastchunkidtype;
+
+ /* Nothing to report without a TOAST table */
+ if (!OidIsValid(toastrelid))
+ return InvalidOid;
+
+ typid = get_atttype(toastrelid, 1);
+ if (!OidIsValid(typid))
+ elog(ERROR, "cache lookup failed for TOAST relation %u",
+ toastrelid);
+ if (typid != OIDOID && typid != OID8OID)
+ elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u",
+ typid, toastrelid);
+
+ relation->rd_toastchunkidtype = typid;
+ return typid;
+}
+
/*
* RelationGetIndexExpressions -- get the index expressions for an index
*
@@ -6530,6 +6566,7 @@ load_relcache_init_file(bool shared)
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
rel->rd_droppedSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_toastchunkidtype = InvalidOid;
rel->pgstat_info = NULL;
/*
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 21bd67c11ea0..83de016eaa58 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,8 +417,10 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
-so that four chunk rows will fit on a page, making it about 2000 bytes).
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default these values are chosen
+so that four chunk rows will fit on a page, making them about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
<acronym>TOAST</acronym> table has the columns
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the
+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index c89fab8764ac..edecbd792b0c 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -74,7 +74,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1565,9 +1567,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 max_chunk_size;
Oid8 toast_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ toast_valueid = ta->va_valueid;
+
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1671,8 +1676,11 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
+ toast_external_data toast_ext_data;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1731,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1775,27 +1783,28 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
- /*
- * Must copy attr into toast_pointer for alignment considerations
- */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ /* Must copy attr into a decoded pointer for alignment considerations */
+ toast_external_info_get(attr, &toast_ext_data);
+ va_tag_value = toast_ext_data.tag;
+ toast_pointer_valueid = toast_ext_data.valueid;
+ va_rawsize = toast_ext_data.rawsize;
+ va_extinfo = toast_ext_data.extinfo;
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1849,7 +1858,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ /* The pointer has already been decoded above, just reuse it */
+ ta->tag = va_tag_value;
+ ta->va_valueid = toast_pointer_valueid;
+ ta->va_extinfo = va_extinfo;
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1879,17 +1891,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
Oid8 toast_valueid;
Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = VARATT_EXTINFO_GET_EXTSIZE(ta->va_extinfo);
+ toast_typid = (ta->tag == VARTAG_ONDISK_OID8) ? OID8OID : OIDOID;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid,
- ta->toast_pointer.va_valueid);
+ toast_valueid);
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1909,8 +1922,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index de21cea65f96..9985c8b21e31 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4358,6 +4358,7 @@ timeout_params
timerCA
tlist_vinfo
toast_compress_header
+toast_external_data
tokenize_error_callback_arg
transferMode
transfer_thread_arg
@@ -4414,6 +4415,7 @@ vacuumingOptions
validate_string_relopt
varatt_expanded
varatt_external_oid
+varatt_external_oid8
varatt_indirect
varattrib_1b
varattrib_1b_e
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 15:44 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
2 siblings, 2 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-11 15:44 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Fri, Sep 11, 2026 at 12:33 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> > 7/ I think we can deduplicate most of the code to reduce the if (OID8)
> > else (OID) branching. I tried to do so and attached a diff on top of
> > v16-0007. Please have a look.
> > 8/ Also, the code in toast_save_datum() now looks a bit complicated
> > and duplicated, and the comment about the race condition during
> > rewrite sits only in the OID8 block, which applies to both. I tried to
> > deduplicate it by moving the rewrite block to a separate function in
> > the attached diff. Please have a look.
>
> OK, that's a big chunk that reduces by close to 30% my previous bigger
> chunk of code.
>
> Finally, some numbers for the last patch that introduces the new
> vartag (with the biggest attribute fix included in last patch):
> - v17:
> 16 files changed, 430 insertions(+), 212 deletions(-)
> - v16:
> 15 files changed, 661 insertions(+), 219 deletions(-)
>
> In short you are cutting 240 lines of code for the last changes with
> your suggestions, and make the code much more readable. I'd say that
> this is nice.
Nice!
> Attached is a rebased v17, with the last 5 patches and your
> refactorings integrated in a cleaner manner, based on my points from
> above.
I reviewed v17 patches, overall they look good to me.
v17-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch: It
looks good to me. No difference from that of v16 or v15 (the versions
I previously reviewed).
v17-0002-Switch-pg_column_toast_chunk_id-return-value-fro.patch: It
looks good to me. No difference from that of v16 or v15 (the versions
I previously reviewed).
v17-0003-Add-support-for-oid8-TOAST-values.patch: It looks good to me
with one nit. Since chunk_id is always at attnum = 1 and all the
toast_valueid_scankey_init() callers pass it as 1, and this init
function is just to fetch the chunk_id, can we just hard-code it
inside and remove the attnum function parameter?
v17-0004-Add-battery-of-tests-related-oid8.patch: It looks good to me.
No difference from that of v16 or v15 (the versions I previously
reviewed). pg_dump/pg_restore for demoing OID4 to OID8 migration for
existing tables and TOAST table tests during pg_upgrade could be
follow-up patches. This makes me think, if someone does an OID8 to
OID4 migration and the chunk_ids are beyond the 4-billion limit, the
restore should fail rather than silently wrapping the chunk_ids and
causing TOAST index insert failures. My point is, we need to test this
case as well.
v17-0005-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 16:03 Marcos Pegoraro <marcos@f10.com.br>
parent: Michael Paquier <michael@paquier.xyz>
2 siblings, 1 reply; 72+ messages in thread
From: Marcos Pegoraro @ 2026-09-11 16:03 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Em sex., 11 de set. de 2026 às 04:33, Michael Paquier <michael@paquier.xyz>
escreveu:
> Thanks again for all the reviews.
>
Did you forget to mention the new REPACK command on create_table.SGML file ?
<acronym>TOAST</acronym> relation, and a table rewrite such as
<link linkend="sql-cluster"><command>CLUSTER</command></link> or
<link linkend="sql-vacuum"><command>VACUUM FULL</command></link>
preserves the type already in use. A dump and restore, on the other
regards
Marcos
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 17:07 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
1 sibling, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-11 17:07 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Fri, Sep 11, 2026 at 8:44 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Fri, Sep 11, 2026 at 12:33 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > > 7/ I think we can deduplicate most of the code to reduce the if (OID8)
> > > else (OID) branching. I tried to do so and attached a diff on top of
> > > v16-0007. Please have a look.
> > > 8/ Also, the code in toast_save_datum() now looks a bit complicated
> > > and duplicated, and the comment about the race condition during
> > > rewrite sits only in the OID8 block, which applies to both. I tried to
> > > deduplicate it by moving the rewrite block to a separate function in
> > > the attached diff. Please have a look.
> >
> > OK, that's a big chunk that reduces by close to 30% my previous bigger
> > chunk of code.
> >
> > Finally, some numbers for the last patch that introduces the new
> > vartag (with the biggest attribute fix included in last patch):
> > - v17:
> > 16 files changed, 430 insertions(+), 212 deletions(-)
> > - v16:
> > 15 files changed, 661 insertions(+), 219 deletions(-)
> >
> > In short you are cutting 240 lines of code for the last changes with
> > your suggestions, and make the code much more readable. I'd say that
> > this is nice.
>
> Nice!
>
> > Attached is a rebased v17, with the last 5 patches and your
> > refactorings integrated in a cleaner manner, based on my points from
> > above.
>
> I reviewed v17 patches, overall they look good to me.
>
> v17-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch: It
> looks good to me. No difference from that of v16 or v15 (the versions
> I previously reviewed).
>
> v17-0002-Switch-pg_column_toast_chunk_id-return-value-fro.patch: It
> looks good to me. No difference from that of v16 or v15 (the versions
> I previously reviewed).
>
> v17-0003-Add-support-for-oid8-TOAST-values.patch: It looks good to me
> with one nit. Since chunk_id is always at attnum = 1 and all the
> toast_valueid_scankey_init() callers pass it as 1, and this init
> function is just to fetch the chunk_id, can we just hard-code it
> inside and remove the attnum function parameter?
>
> v17-0004-Add-battery-of-tests-related-oid8.patch: It looks good to me.
> No difference from that of v16 or v15 (the versions I previously
> reviewed). pg_dump/pg_restore for demoing OID4 to OID8 migration for
> existing tables and TOAST table tests during pg_upgrade could be
> follow-up patches. This makes me think, if someone does an OID8 to
> OID4 migration and the chunk_ids are beyond the 4-billion limit, the
> restore should fail rather than silently wrapping the chunk_ids and
> causing TOAST index insert failures. My point is, we need to test this
> case as well.
>
> v17-0005-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
Upon thinking more, I think we need a test case where pg_resetwal sets
the OID8 counter beyond the OID4 limit of 4 billion and verify that
the TOAST chunk_id goes beyond that to properly cover the OID8 case.
The tests added by v17-0004-Add-battery-of-tests-related-oid8.patch,
although they create a table with OID8 TOAST, all use chunk_ids within
the OID4 limits.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 17:38 Rui Zhao <zhaorui126@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
2 siblings, 1 reply; 72+ messages in thread
From: Rui Zhao @ 2026-09-11 17:38 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi Michael,
v17 applies to master (6a1c1102c4), builds warning-free, and make
check-world passes here. Three small patches attached: 0001 applies to
master, 0002 and 0003 on top of 0005.
1. The 8-byte counter survives standby promotion and crash recovery,
which the tests of cb298616463d do not cover; 0001 adds the standby case
to 004_timeline_switch.pl.
Primary counter set to 4295067296 (2^32 + 100000), base backup, then
20000 values into an oid8 table, so the standby sees the counter only
through XLOG_NEXTOID records. After promotion and 3000 more values,
next_oid is 4295100064; the same after killing the primary with an
immediate shutdown and restarting it. All 23000 values read back equal
on both nodes, chunk_ids 4295067301 to 4295094871, all distinct.
2. pg_upgrade works both ways that matter: a v17 cluster with oid8
tables and the counter past 2^32 comes out usable, and a 16beta2 cluster
comes out with its oid TOAST tables intact and able to add oid8 ones.
v17 to v17: NextOID 4295067406 before and after, then 120 more rows into
an oid8 and an oid table, 120 new distinct IDs each. 16beta2 to v17: an
external and a compressed TOAST table stay oid, reads and inserts work,
verify_heapam() finds nothing, an oid8 table created next to them works,
and a plain dump restores with the same chunk_id types.
3. Logical decoding of oid8 values past 2^32 works, but no test in 0004
would catch it if it broke; 0002 adds one that does.
Publisher and subscriber in one cluster, counter past 2^32, two rows with
9000-byte values, one then updated on a non-TOAST column, one given a
new value: the subscriber ends with the same lengths and md5s, chunk_ids
4295067317 and 4295067319. The toast.sql additions in 0004 run with a
young counter, so their IDs fit in 4 bytes and a chunk_id read as an Oid
still matches. 0002 moves the counter past 2^32 in
006_logical_decoding.pl and decodes one oid8 value; with the chunk_id
read the way it was before v14 the row comes out as
data[text]:unchanged-toast-datum, with v17 it passes.
4. A rewrite after ALTER TABLE SET (toast_value_type) keeps the TOAST
table's type either way, as Bharath concluded from the code.
VACUUM FULL, CLUSTER, REPACK, SET UNLOGGED and SET LOGGED, on an oid
table switched to oid8 and on an oid8 table with IDs past 2^32 switched
to oid: chunk_id type unchanged, 50 of 50 values intact, verify_heapam()
and bt_index_check() clean after each. Only a TOAST table created
afterwards (ADD COLUMN on a table that had none, CREATE TABLE AS) picks
up the new value.
5. For oid8 tables the -o docs are incomplete: a lost pg_control moves
the counter backwards by itself, and the value to set, which the docs
say there is no easy way to find, is one query away. 0003 says so under
-o, and adds that the reloption has to be given on every partition (the
reloptions test checks that a partitioned table rejects it; I checked
that a partition created without it gets an oid TOAST table).
Rare case: pg_control lost, cluster brought back with pg_resetwal -f as
documented, the OID counter comes back as the guess, 10000. On a fresh
cluster with 1000 values in an oid8 table (chunk_ids 16394 to 17393) and
1000 in an oid table, the oid table takes new rows as before and inserts
into the oid8 table fail with
ERROR: duplicate key value violates unique constraint "pg_toast_16384_index"
each failed statement moving the counter by one, so a thousand failures
before it comes back on its own. SELECT max(chunk_id) on the oid8 TOAST
table gave 17393; pg_resetwal -o 17394 put it back to work.
Regards,
Rui
Attachments: download all (3 files, .tar.gz)
0001-Check-that-a-promoted-standby-keeps-the-8-byte-OID-c.patch (application/octet-stream, 2.0K) download
From 9e7e855d7d84d687108bfbf0d64c0591e4a69cf1 Mon Sep 17 00:00:00 2001
From: Rui Zhao <zhaorui126@gmail.com>
Date: Thu, 10 Sep 2026 10:16:56 +0800
Subject: [PATCH 1/3] Check that a promoted standby keeps the 8-byte OID
counter
The counter reaches a standby through XLOG_NEXTOID and checkpoint
records. Move it past 2^32 on the primary of the timeline-switch test
and read it back from the promoted standby.
---
src/test/recovery/t/004_timeline_switch.pl | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
index 4bf111346e..4085d7d04d 100644
--- a/src/test/recovery/t/004_timeline_switch.pl
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -14,6 +14,13 @@ use Test::More;
# Initialize primary node
my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(allows_streaming => 1);
+
+# Move the OID counter past 2^32, to check below that the promoted standby
+# receives its 8-byte value through WAL.
+my $next_oid = '4295067296'; # 2^32 + 100000
+command_ok(
+ [ 'pg_resetwal', '--next-oid' => $next_oid, $node_primary->data_dir ],
+ 'set an 8-byte OID counter on the primary');
$node_primary->start;
# Take backup
@@ -51,6 +58,16 @@ $node_standby_1->psql(
stdout => \$psql_out);
is($psql_out, 't', "promotion of standby with pg_promote");
+# The OID counter of the promoted standby was set by replaying the
+# XLOG_NEXTOID and checkpoint records of the primary. Force a checkpoint so
+# that pg_control_checkpoint() reports the value in use.
+$node_standby_1->safe_psql('postgres', 'CHECKPOINT');
+is( $node_standby_1->safe_psql(
+ 'postgres',
+ "SELECT next_oid >= '$next_oid'::oid8 FROM pg_control_checkpoint()"),
+ 't',
+ '8-byte OID counter carried over to the promoted standby');
+
# Switch standby 2 to replay from standby 1. During the timeline switch,
# the WAL receiver process on standby 2 should not be stopped, and the
# new primary connection string should not be visible
--
2.43.7
0002-Decode-an-oid8-TOAST-value-with-an-ID-past-2-32.patch (application/octet-stream, 2.5K) download
From 506d953a8c286c489153fbb52515507f86a1a275 Mon Sep 17 00:00:00 2001
From: Rui Zhao <zhaorui126@gmail.com>
Date: Fri, 11 Sep 2026 00:29:11 +0800
Subject: [PATCH 2/3] Decode an oid8 TOAST value with an ID past 2^32
The test_decoding tests run with a young OID counter, so the value IDs
they see fit in 4 bytes and a chunk_id read as an Oid still matches.
Move the counter past 2^32 in the logical decoding TAP test and decode
one oid8 value there.
---
src/test/recovery/t/006_logical_decoding.pl | 33 +++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 97d11f98b5..57d3cf91ee 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -20,6 +20,13 @@ $node_primary->append_conf(
'postgresql.conf', qq(
wal_level = logical
));
+
+# Move the OID counter past 2^32, so that the oid8 TOAST values decoded at
+# the end of this test have IDs that do not fit in 4 bytes.
+my $next_oid = '4295067296'; # 2^32 + 100000
+command_ok(
+ [ 'pg_resetwal', '--next-oid' => $next_oid, $node_primary->data_dir ],
+ 'set an 8-byte OID counter');
$node_primary->start;
$node_primary->safe_psql('postgres',
@@ -275,6 +282,32 @@ is( $node_primary->safe_psql(
qq(Check that reset timestamp is later after resetting stats for slot '$stats_test_slot1' again.)
);
+# Decode an oid8 TOAST value whose ID is past 2^32. The reorder buffer
+# collects the chunks by chunk_id, and has to read that column with the
+# TOAST table's type, or the value is reported as unchanged.
+$node_primary->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE toasted_oid8 (id int PRIMARY KEY, data text)
+ WITH (toast_value_type = 'oid8');
+ ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+ SELECT pg_create_logical_replication_slot('oid8_slot', 'test_decoding');
+ INSERT INTO toasted_oid8 VALUES (1, repeat('1234567890', 2000));
+]);
+is( $node_primary->safe_psql(
+ 'postgres',
+ "SELECT pg_column_toast_chunk_id(data) > '$next_oid'::oid8 FROM toasted_oid8"
+ ),
+ 't',
+ 'oid8 TOAST value has an ID past 2^32');
+$result = $node_primary->safe_psql(
+ 'postgres',
+ "SELECT data FROM pg_logical_slot_get_changes('oid8_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')"
+);
+like(
+ $result,
+ qr/data\[text\]:'(?:1234567890){2000}'/,
+ 'oid8 TOAST value past 2^32 is decoded in full');
+
# done with the node
$node_primary->stop;
--
2.43.7
0003-Document-two-things-about-toast_value_type-oid8.patch (application/octet-stream, 2.6K) download
From 6313ebe09f3fe0e68ee5cb9f3eb25a23ad28a621 Mon Sep 17 00:00:00 2001
From: Rui Zhao <zhaorui126@gmail.com>
Date: Fri, 11 Sep 2026 00:35:47 +0800
Subject: [PATCH] Document two things about toast_value_type = oid8
A partitioned table rejects the parameter like any other storage
parameter, and a partition does not inherit it from its parent; say so
next to the parameter.
An oid8 TOAST table takes the OID counter's value without checking the
index, so a counter moved below a chunk_id in use makes inserts fail.
Say so under pg_resetwal -o, with the way to find a safe value.
---
doc/src/sgml/ref/create_table.sgml | 7 +++++++
doc/src/sgml/ref/pg_resetwal.sgml | 12 ++++++++++++
2 files changed, 19 insertions(+)
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 6f9a87ef44..3de92ffeed 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1679,6 +1679,13 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
hand, recreates the <acronym>TOAST</acronym> relation and therefore
applies the current value of this parameter.
</para>
+ <para>
+ As with any storage parameter, this cannot be set on a partitioned
+ table, and a partition does not inherit it from its parent: a
+ partition created without it gets a <acronym>TOAST</acronym> relation
+ with an <type>oid</type> <literal>chunk_id</literal>. Specify it on
+ each partition that needs it.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index d34aa26413..02678eb56f 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -300,6 +300,18 @@ PostgreSQL documentation
backwards. The counter is 8 bytes wide and never wraps around during
the life of a cluster.
</para>
+
+ <para>
+ Tables with <xref linkend="reloption-toast-value-type"/> set to
+ <literal>oid8</literal> are the exception: their
+ <acronym>TOAST</acronym> values take the counter's value as is, so a
+ counter set below a <structfield>chunk_id</structfield> already in use
+ makes inserts into such a table fail with duplicate key errors, and
+ each failure moves the counter forward by only one. For these tables
+ the largest value in use is easy to find, with
+ <literal>SELECT max(chunk_id)</literal> on each of their
+ <acronym>TOAST</acronym> tables; set the next OID above it.
+ </para>
</listitem>
</varlistentry>
--
2.43.7
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-11 23:30 Michael Paquier <michael@paquier.xyz>
parent: Rui Zhao <zhaorui126@gmail.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-11 23:30 UTC (permalink / raw)
To: Rui Zhao <zhaorui126@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sat, Sep 12, 2026 at 01:38:06AM +0800, Rui Zhao wrote:
> v17 applies to master (6a1c1102c4), builds warning-free, and make
> check-world passes here. Three small patches attached: 0001 applies to
> master, 0002 and 0003 on top of 0005.
Thanks for looking and testing.
0001 and 0002 sound like good ideas in terms of coverage, why not, but
I don't really buy the benefit offered by 0003 (see below for reason).
> Rare case: pg_control lost, cluster brought back with pg_resetwal -f as
> documented, the OID counter comes back as the guess, 10000. On a fresh
> cluster with 1000 values in an oid8 table (chunk_ids 16394 to 17393) and
> 1000 in an oid table, the oid table takes new rows as before and inserts
> into the oid8 table fail with
If you have "lost" a control file, an oid8 value in the past is the
least of your worries. At least this can be safely be set: just move
it to something like (UINT64_MAX / N) and you should be good enough, N
being a natural integer of your choice. I still don't buy that adding
a re-check of the value in the TOAST insert path is worth the cost of
this case.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-12 17:46 Rui Zhao <zhaorui126@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 72+ messages in thread
From: Rui Zhao @ 2026-09-12 17:46 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi Michael,
> I still don't buy that adding a re-check of the value in the TOAST
> insert path is worth the cost of this case.
Agreed, and 0003 does not add one; it is two paragraphs of documentation.
The one under pg_resetwal -o can go. The other one, in create_table.sgml,
says that the parameter cannot be set on a partitioned table, that a
partition does not inherit it, and that a partition created without it
gets an oid TOAST table. That has nothing to do with the counter; keep it
or drop it as you see fit.
Regards,
Rui
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-13 16:22 Oleg Bartunov <obartunov@postgrespro.ru>
parent: Michael Paquier <michael@paquier.xyz>
2 siblings, 1 reply; 72+ messages in thread
From: Oleg Bartunov @ 2026-09-13 16:22 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
Hi, Michael !
While looking at v16-0007, one thing looked suspicious to me.
VARATT_IS_EXTERNAL_ONDISK() used to effectively mean that the datum
could be interpreted as the existing OID-based external pointer. With
OID8 it becomes a family predicate, accepting both OID and OID8
pointers.
This looked like a potentially silent compatibility problem for
extensions: old code may still compile, but its assumption about the
pointer layout is no longer true.
I did a quick check of out-of-tree users and found at least two
examples. PostGIS has code like this in ToastCacheGetGeometry():
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
...
struct varatt_external ve;
VARATT_EXTERNAL_GET_POINTER(ve, attr);
Oid valueid = ve.va_valueid;
Oid toastrelid = ve.va_toastrelid;
MobilityDB carries the same pattern. So an OID8 datum can now pass
VARATT_IS_EXTERNAL_ONDISK() and existing code can then interpret it
using the old OID pointer layout.
I noticed this because we have been experimenting with alternative
physical representations of large varlena values and tracing which
parts of the value lifecycle actually depend on a particular TOAST
representation. OID8 turned out to be a useful real-world test of that
boundary.
Perhaps we need to distinguish explicitly between "any on-disk
external pointer" and "the old OID on-disk pointer", rather than
changing the meaning of the existing predicate?
-Oleg
On Wed, Sep 9, 2026 at 6:43 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Tue, Sep 08, 2026 at 04:55:16PM -0700, Bharath Rupireddy wrote:
> > I don't think pg_control_checkpoint() and pg_controldata emitting an
> > OID extracted from the OID8 value is the right idea. I understand that
> > tools and external modules do need to adapt to the wider value, but
> > that is fine for a new major version, and the control file version
> > bump gives them the signal to do so. Also, the extra 4 bytes from OID8
> > are absorbed by existing padding, so the control file stays well under
> > the safe size limit of 512 bytes anyway.
>
> I don't mean to extend the on-disk file with an extra 4-bytes for the
> Oid. I mean to *show* an extra field based on a casted value. These
> are two different things.
>
> > v15-0001-Enlarge-OID-generation-to-8-bytes.patch: this looks good to
> > me, with one comment. Although the wraparound with an 8-byte OID is
> > hundreds of years away, and 0001 assumes it never happens, do we need
> > to handle the case where someone sets the OID counter closer to the
> > 8-byte OID limit using pg_resetwal and still hits the wraparound? We
> > can either reject this in pg_resetwal, handle the wraparound case in
> > the caller, document it, or mention it in the commit message or
> > somewhere near pg_resetwal. This is not a blocker for getting 0001 in,
> > just to be clear. I think this needs to be fixed as part of
> > v15-0006-Add-support-for-oid8-TOAST-values.patch, which does not check
> > for an existing TOAST OID.
>
> A few things worth noting: we have in xlog_redo() and varsup.h a
> couple of comments that mention wraparound without mentioning the oid8
> bits that can bypass it. These need to be slightly adjusted..
>
> I can fancy you with one more case, because there is no entry recheck
> in the TOAST insert path when using an 8-byte OID:
> - Insert some TOAST entries with toast_value_type=oid8.
> - Reset to a past value with pg_resetwal.
> - Reinsert, failing hard on INSERT due to a duplicated key.
>
> My best answer is don't set the counter to the past. Somebody using
> pg_resetwal is looking for trouble already. It's true that not being
> able to handle the OID8 value conflicts like the normal TOAST path
> could be seen as a defect of the OID8 path, but adding an index check
> (while doable of course), is just wasting resources with 8-bytes,
> because in practice that's never hit. So "don't set it to the past"
> seems like an answer good enough? That sounds to me like a
> documentation change.
>
> > v15-0003-Switch-pg_column_toast_chunk_id-return-value-fro.patch: this
> > looks good to me, with one comment. The chunk_id cast in
> > cluster-toast-value-reuse.spec seems redundant, because that column
> > comes from the output of pg_column_toast_chunk_id(). In
> > misc_functions.sql it is needed. This makes me think about whether we
> > need to support OID to OID8 comparison in more cases: existing queries
> > and monitoring tools that do pg_column_toast_chunk_id(x) = some_oid,
> > or that store the result in an OID column, will now get a
> > type-mismatch error and have to add an explicit cast. Could we ease
> > that by making the OID to OID8 cast implicit, so that an OID compared
> > against an OID8 is promoted automatically? This is not a blocker for
> > getting 0003 in, just to be clear.
>
> My reply here would be to cast the some_oid to oid8, or invent a
> secondary oid8 specific function, leaving pg_column_toast_chunk_id()
> alone and failing if trying to use it with an oid8 TOAST table?
>
> I suspect that there are not that many users of
> pg_column_toast_chunk_id() anyway. May be wrong of course, but a cast
> to oid8 is fine by me. I'm just not sure that it is worth the churn
> of having a second function with a different return type.
>
> > v15-0004-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch: this
> > looks good to me, with a nit. Use
> > format_type_be(binary_upgrade_next_toast_chunk_id_typoid) instead of
> > %u for better readability in the "cannot support toast chunk_id type"
> > error message.
>
> Not sure it matter. That should not be reachable in practice.
>
> > v15-0005-Add-relation-option-toast_value_type.patch: some comments.
> > 1/ Parenthesize defaulttarg.
> > 2/ Nit: double space after "creating".
>
> Noted these down for later.
>
> > 3/ I understand that v15-0008 adds more tests, but can we have simple
> > tests in this patch as well, like creating a table with
> > toast_value_type='oid' and with an unknown type to verify the error?
> > This would make this patch independent.
>
> Yeah, I guess that you're right to complain here. Most of the
> reloption tests could be moved to the patch that introduces the
> reloption, for clarity.
>
> > 5/ Following on from (4). Say I create a table with oid, insert data,
> > change the reloption to oid8, and then run a rewrite (VACUUM FULL,
> > CLUSTER or REPACK). The final table still uses oid, because the
> > rewrite preserves the existing TOAST table by swapping contents rather
> > than recreating it. But the transient table that make_new_heap()
> > builds does get created with the new type, because it goes through
> > create_toast_table(), which reads the reloption without checking
> > whether this is a rewrite. With oid this makes no difference, but once
> > oid8 is added the transient TOAST table would be oid8 while the
> > existing TOAST table still says oid. Is that expected?
>
> I am not completely sure to follow here? Wouldn't that be the case of
> create_toast_table() defining a OIDOldToast. In this case, we inherit
> the type ID from the old table when creating what would be the
> transient table. See the comment named as "break the world" in
> create_toast_table() added in the patch. Or perhaps I'm missing your
> point entirely.
>
> > 1/ Measured the additional storage space and impact on query
> > performance for the same TOAST limits under 4 billion (OID limit).
> >
> > At 4 GB TOAST (500k rows × 8 KB), oid8 costs +40% in TOAST index size
> > and +15% in heap size, with no measurable change to the TOAST table
> > size or full-detoast scan time. Queries [1]. The heap increase comes
> > from the on-disk TOAST pointer growing 4 bytes and the tuple
> > alignment. The TOAST index grows for the same reason on its chunk_id
> > key. The TOAST table size stays about the same because the extra
> > chunk_id bytes are small next to the roughly 2 KB of chunk_data and do
> > not change how many chunks fit per page.
>
> Yeah, the cost is part of the game here..
>
> > 2/ Used oid8 for all tables by default and ran regression tests to see
> > if there are any issues.
>
> Did this one as well at some point. I'm pretty sure I've caught some
> of the rewrite bugs with that, but I don't recall entirely..
>
> > [3]
> > doesn't match '(?^:(?^ms:heap table "postgres\.public\.test", block 0,
> > offset 5:\s+)data begins at offset 152 beyond the tuple length 58)'
> > # Looks like you failed 1 test of 32.
> > t/004_verify_heapam.pl ... Dubious, test returned 1 (wstat 256, 0x100)
> > Failed 1/32 subtests
>
> This needs manipulation of on-disk data. I have done that at some
> point but I did not see that as worth the extra cost. Having tests
> for the valid cases sounded more than enough, because we care about
> making sure that the vartags are passed around right. Whether they
> actual report a corruption is less interesting.
>
> Before reworking on a new rebase of the whole patch set, attached is
> an adjusted 0001 for the 8-byte counter, with adjustments for the docs
> of pg_resetwal and fixes for a few comments that worry about OID
> wraparound, that I have noticed after more review.
>
> How does this part look?
> --
> Michael
--
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-13 23:17 Michael Paquier <michael@paquier.xyz>
parent: Oleg Bartunov <obartunov@postgrespro.ru>
0 siblings, 2 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-13 23:17 UTC (permalink / raw)
To: Oleg Bartunov <obartunov@postgrespro.ru>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 13, 2026 at 10:07:27PM +0545, Oleg Bartunov wrote:
> I did a quick check of out-of-tree users and found at least two
> examples. PostGIS has code like this in ToastCacheGetGeometry():
postgis holds a direct reference to varatt_external in its
libpgcommon/lwgeom_cache.c, which is wanted to feed data to
ToastCacheArgument. As far as I can see, their change is not
complicated, upgrade ToastCacheArgument.valueid to be an Oid8, then,
instead of the direct use of varatt_external() switch to a
toast_external_data combined with a call of toast_external_info_get()
to make the switch transparent, so as the TOAST value, Oid or Oid8 can
be retrieved.
So it's not a complicated change, and as far as I can see it's mostly
a matter of two PG_VERSION_NUM blocks plus a few tests. Using
varatt_external in the code would lead to a compilation failure,
because it's now renamed varatt_external_oid, one will not see a
runtime failure. It's going to be hard to miss that something is
wrong in this case.
> MobilityDB carries the same pattern. So an OID8 datum can now pass
> VARATT_IS_EXTERNAL_ONDISK() and existing code can then interpret it
> using the old OID pointer layout.
EXTERNAL_ONDISK() means either of them.
> I noticed this because we have been experimenting with alternative
> physical representations of large varlena values and tracing which
> parts of the value lifecycle actually depend on a particular TOAST
> representation. OID8 turned out to be a useful real-world test of that
> boundary.
MobilityDB is carrying the same patterns because it uses a copy-paste
of the postgis code. Again, this uses directly varatt_external, which
would fail at compilation under PG20. It sounds to me that they just
update their code on a regular basis based on postgis, so it means
that sending only a patch to adjust postgis would be enough. I'm
happy to take some time to do that, it
> Perhaps we need to distinguish explicitly between "any on-disk
> external pointer" and "the old OID on-disk pointer", rather than
> changing the meaning of the existing predicate?
It seems to me that this difference already exists in the patch, due
to the rebranding of varatt_external.
Note that if somebody has the idea to use VARTAG_SIZE() to copy
external varlena pointers, then the change is transparent.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-13 23:51 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-13 23:51 UTC (permalink / raw)
To: Oleg Bartunov <obartunov@postgrespro.ru>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 14, 2026 at 08:17:47AM +0900, Michael Paquier wrote:
> MobilityDB is carrying the same patterns because it uses a copy-paste
> of the postgis code. Again, this uses directly varatt_external, which
> would fail at compilation under PG20. It sounds to me that they just
> update their code on a regular basis based on postgis, so it means
> that sending only a patch to adjust postgis would be enough. I'm
> happy to take some time to do that, it
This got cut mid-sentence, it's just without the extra part:
"I'm happy to make some time to do that."
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 00:46 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-14 00:46 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Sep 11, 2026 at 10:07:37AM -0700, Bharath Rupireddy wrote:
> Upon thinking more, I think we need a test case where pg_resetwal sets
> the OID8 counter beyond the OID4 limit of 4 billion and verify that
> the TOAST chunk_id goes beyond that to properly cover the OID8 case.
> The tests added by v17-0004-Add-battery-of-tests-related-oid8.patch,
> although they create a table with OID8 TOAST, all use chunk_ids within
> the OID4 limits.
Yes, Rui has proposed a few things to cross the gap here:
https://www.postgresql.org/message-id/CAHWVJhGkb1n_QSS4M=PKJD5HHcwK3KXpFwVxoRVLzAG3j+2fhQ@mail.gmail...
One thing that he has done is to check a reorderbuffer change with the
control file holding a value larger than UINT32_MAX. I agree that
having a "normal" SQL case would be adapted, but why not just plug one
SELECT into pg_upgrade once varatt_external_oid8 is introduced? We
hold into toasttest_oid8 and toasttest_oid in patch 0004 for this
purpose. The check cannot be added before 0005, but the basics are
here.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 01:39 Oleg Bartunov <obartunov@postgrespro.ru>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 72+ messages in thread
From: Oleg Bartunov @ 2026-09-14 01:39 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
<div dir='auto'><div><br><div><br><div class="elided-text">On Sep 14, 2026 05:02, Michael Paquier <michael@paquier.xyz> wrote:<br type="attribution"><blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">On Sun, Sep 13, 2026 at 10:07:27PM +0545, Oleg Bartunov wrote:
<br>
> I did a quick check of out-of-tree users and found at least two
<br>
> examples. PostGIS has code like this in ToastCacheGetGeometry():
<br>
<br>
postgis holds a direct reference to varatt_external in its
<br>
libpgcommon/lwgeom_cache.c, which is wanted to feed data to
<br>
ToastCacheArgument. As far as I can see, their change is not
<br>
complicated, upgrade ToastCacheArgument.valueid to be an Oid8, then,
<br>
instead of the direct use of varatt_external() switch to a
<br>
toast_external_data combined with a call of toast_external_info_get()
<br>
to make the switch transparent, so as the TOAST value, Oid or Oid8 can
<br>
be retrieved.
<br>
<br>
So it's not a complicated change, and as far as I can see it's mostly
<br>
a matter of two PG_VERSION_NUM blocks plus a few tests. Using
<br>
varatt_external in the code would lead to a compilation failure,
<br>
because it's now renamed varatt_external_oid, one will not see a
<br>
runtime failure. It's going to be hard to miss that something is
<br>
wrong in this case.
<br>
<br>
> MobilityDB carries the same pattern. So an OID8 datum can now pass
<br>
> VARATT_IS_EXTERNAL_ONDISK() and existing code can then interpret it
<br>
> using the old OID pointer layout.
<br>
<br>
EXTERNAL_ONDISK() means either of them.
<br>
<br>
> I noticed this because we have been experimenting with alternative
<br>
> physical representations of large varlena values and tracing which
<br>
> parts of the value lifecycle actually depend on a particular TOAST
<br>
> representation. OID8 turned out to be a useful real-world test of that
<br>
> boundary.
<br>
<br>
MobilityDB is carrying the same patterns because it uses a copy-paste
<br>
of the postgis code. Again, this uses directly varatt_external, which
<br>
would fail at compilation under PG20. It sounds to me that they just
<br>
update their code on a regular basis based on postgis, so it means
<br>
that sending only a patch to adjust postgis would be enough. I'm
<br>
happy to take some time to do that, it</p></blockquote></div></div></div><div dir="auto">Thanks, that makes sense. I had missed that the varatt_external rename turns this particular case into a compile-time failure.</div><div dir="auto"><div><div class="elided-text"><blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr"><br>
> Perhaps we need to distinguish explicitly between "any on-disk
<br>
> external pointer" and "the old OID on-disk pointer", rather than
<br>
> changing the meaning of the existing predicate?
<br>
<br>
It seems to me that this difference already exists in the patch, due
<br>
to the rebranding of varatt_external.
<br>
<br>
Note that if somebody has the idea to use VARTAG_SIZE() to copy
<br>
external varlena pointers, then the change is transparent.</p></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">Just to make sure I understand the intended abstraction correctly: with this patch, should I think of VARATT_IS_EXTERNAL_ONDISK() as identifying a family of on-disk external representations, while the vartag identifies the concrete physical representation within that family?</div><div dir="auto">In other words:</div><div dir="auto">VARATT_IS_EXTERNAL_ONDISK() -> representation family, and </div><div dir="auto">VARTAG_* -> concrete representation</div><div dir="auto">If so, that is actually the distinction I was looking for.</div><div dir="auto"><br></div><div dir="auto">Oleg </div><div dir="auto"><div><div class="elided-text"><blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">
<br>
--
<br>
Michael
<br>
</p>
</blockquote></div><br></div></div></div>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 02:16 Michael Paquier <michael@paquier.xyz>
parent: Oleg Bartunov <obartunov@postgrespro.ru>
0 siblings, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-14 02:16 UTC (permalink / raw)
To: Oleg Bartunov <obartunov@postgrespro.ru>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; Postgres hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 14, 2026 at 07:24:34AM +0545, Oleg Bartunov wrote:
> Just to make sure I understand the intended abstraction correctly:
> with this patch, should I think of VARATT_IS_EXTERNAL_ONDISK() as
> identifying a family of on-disk external representations, while the
> vartag identifies the concrete physical representation within that
> family? In other words: VARATT_IS_EXTERNAL_ONDISK() ->
> representation family, and VARTAG_* -> concrete representation If
> so, that is actually the distinction I was looking for.
(Perhaps avoid html..)
In this context, VARATT_IS_EXTERNAL_ONDISK() still means "Is this
varlena an on-disk external TOAST pointer?", it just means that it
englobes *both* varatt_external_oid and varatt_external_oid8.
I was surprised to see that we can rely on EXTERNAL_ONDISK() heavily
in the core code once we have the second external on-disk vartag, and
more surprised that we don't really need a set of macros of the type
EXTERNAL_ONDISK_OID() or EXTERNAL_ONDISK_OID8() with the changes in
detoast.h suggested by Bharath. We could add these types of macros,
of course, but it's just bloating more varatt.h with stuff we don't
need at the end, at least when it comes to the core code and what I've
seen outside of core in the open. This last remark may not apply to
fancier outsider cases out there I am not aware of.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 05:00 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
1 sibling, 4 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-14 05:00 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Sep 11, 2026 at 08:44:26AM -0700, Bharath Rupireddy wrote:
> I reviewed v17 patches, overall they look good to me.
>
> v17-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch: It
> v17-0002-Switch-pg_column_toast_chunk_id-return-value-fro.patch: It
I have discarded the change of reorderbuffer.c that was in v17-0001,
and added it back to what's now v18-0001. Then applied these two
after an extra round of self-review.
> v17-0003-Add-support-for-oid8-TOAST-values.patch: It looks good to me
> with one nit. Since chunk_id is always at attnum = 1 and all the
> toast_valueid_scankey_init() callers pass it as 1, and this init
> function is just to fetch the chunk_id, can we just hard-code it
> inside and remove the attnum function parameter?
If one wants to control which attribute number to look after, that
could be extended more later on, yes. I have removed the argument
from the function in v18-0001.
> v17-0004-Add-battery-of-tests-related-oid8.patch: It looks good to me.
Okay, thanks.
> pg_dump/pg_restore for demoing OID4 to OID8 migration for
> existing tables and TOAST table tests during pg_upgrade could be
> follow-up patches. This makes me think, if someone does an OID8 to
> OID4 migration and the chunk_ids are beyond the 4-billion limit, the
> restore should fail rather than silently wrapping the chunk_ids and
> causing TOAST index insert failures. My point is, we need to test this
> case as well.
Nah, I don't buy that this is worth the cycles. If one has the idea
to create an oid8 TOAST table, inserts more than 4 billion values into
it, dumps and restores into a table that uses a oid TOAST then the
result is an infinite loop in the backend on restored INSERTs.
Something that I still see as a hole is that we have the tables from
the strings test in the main regression test suite and we could
exploit that a bit more. Hmm. How about resetting the OID in the
control file earlier in the pg_upgrade test and check with a SELECT
post-upgrade that the counters generated by the main regression test
suite are more than 2^32?
Added also some tests for REPACK here in what's now 0002, based on the
feedback of upthread, to check that the a REPACK's rewrite does not
touch the TOAST table definition.
> v17-0005-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
And rebased is a v18 with the three remaining pieces, plus two more
test patches: the tests of Rui with decoding and oid8 values higher
than 2^32, and my test to have the main regression test suite of
pg_upgrade run with a default oid8 in the control file higher than
2^32, checking that assigned chunk numbers are handled correctly.
--
Michael
Attachments: download all (6 files, .tar.gz)
v18-0001-Add-support-for-oid8-TOAST-values.patch (text/plain, 21.5K) download
From 647edfdc6984a9d4f53a2321f138e83c0a36df93 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 14 Sep 2026 13:08:51 +0900
Subject: [PATCH v18 1/5] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, unfortunately casted to OID for now, as we do not have
a vartag_external able to store Oid8 values yet.
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
Note that as of this commit, TOAST table can use oid8 for their chunk_id
attribute, however the data is still stored on the HEAD side using
4-byte wide Oids. A follow-up change that introduces a new
vartag_external will stick 8-byte Oids later.
XXX: Catalog version bump required.
---
src/include/access/toast_internals.h | 4 +
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 3 +-
src/backend/access/common/toast_internals.c | 89 +++++++++++++------
src/backend/access/heap/heaptoast.c | 7 +-
src/backend/catalog/toasting.c | 25 +++++-
.../replication/logical/reorderbuffer.c | 9 +-
src/test/regress/expected/reloptions.out | 15 +++-
src/test/regress/sql/reloptions.sql | 8 ++
doc/src/sgml/ref/create_table.sgml | 5 +-
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 13 +--
13 files changed, 143 insertions(+), 46 deletions(-)
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index bf45889a6428..e03cc1204beb 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -12,6 +12,7 @@
#ifndef TOAST_INTERNALS_H
#define TOAST_INTERNALS_H
+#include "access/skey.h"
#include "access/toast_compression.h"
#include "storage/lockdefs.h"
#include "utils/relcache.h"
@@ -52,6 +53,9 @@ extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
extern Datum toast_save_datum(Relation rel, Datum value,
varlena *oldexternal, uint32 options);
+extern void toast_valueid_scankey_init(ScanKey entry, Oid toast_typid,
+ Oid8 valueid);
+
extern int toast_open_indexes(Relation toastrel,
LOCKMODE lock,
Relation **toastidxs,
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat
index df170b80840b..b84c2bb7a8c3 100644
--- a/src/include/catalog/pg_opclass.dat
+++ b/src/include/catalog/pg_opclass.dat
@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',
+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index a88549dcc555..d61432de705e 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 82491a31a9a4..ea9a04179090 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
@@ -578,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")
+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index ca5ed8dfabca..de4b17fa8164 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,6 +132,7 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
@@ -200,24 +202,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.
+ * Choose a new value to use as the value ID for this toast value, be it
+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But
+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if
+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.
+ * conflict with either new or existing toast value IDs. If the TOAST
+ * table uses 8-byte value IDs, we should not really care much about
+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ }
}
else
{
@@ -263,17 +273,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either
- * old or new toast table
+ * new value; must choose a value that doesn't conflict in either
+ * old or new toast table.
*/
- do
+ if (toast_typid == OID8OID)
+ toast_pointer.va_valueid = GetNewObjectId8();
+ else
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
+ do
+ {
+ toast_pointer.va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ } while (toastid_valueid_exists(rel->rd_toastoid,
+ toast_pointer.va_valueid));
+ }
}
}
@@ -303,7 +318,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ if (toast_typid == OID8OID)
+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ else
+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -366,6 +384,26 @@ toast_save_datum(Relation rel, Datum value,
return PointerGetDatum(result);
}
+/* ----------
+ * toast_valueid_scankey_init -
+ *
+ * Initialize a scan key that matches the value ID column of a TOAST table.
+ * ----------
+ */
+void
+toast_valueid_scankey_init(ScanKey entry, Oid toast_typid, Oid8 valueid)
+{
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
+ if (toast_typid == OID8OID)
+ ScanKeyInit(entry, (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OID8EQ,
+ ObjectId8GetDatum(valueid));
+ else
+ ScanKeyInit(entry, (AttrNumber) 1,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum((Oid) valueid));
+}
+
/* ----------
* toast_delete_datum -
*
@@ -405,10 +443,9 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(toast_pointer.va_valueid));
+ toast_valueid_scankey_init(&toastkey,
+ TupleDescAttr(toastrel->rd_att, 0)->atttypid,
+ toast_pointer.va_valueid);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +492,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +500,12 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ toast_valueid_scankey_init(&toastkey, toast_typid, valueid);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81154c17376c..81d5085ca398 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +657,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(valueid));
+ toast_valueid_scankey_init(&toastkey[0], toast_typid, valueid);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index a4a7b7c91b31..a7079661666c 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -173,6 +174,9 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
case STDRD_OPTION_TOAST_VALUE_TYPE_OID:
toast_chunkid_typid = OIDOID;
break;
+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID8:
+ toast_chunkid_typid = OID8OID;
+ break;
case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID:
elog(ERROR, "unexpected toast_value_type value %d",
value_type);
@@ -210,7 +214,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)
+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&
+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -235,6 +240,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*
+ * Special case here. If OIDOldToast is defined, we need to rely on the
+ * existing table for the job because we do not want to create an
+ * inconsistent relation that would conflict with the parent and break
+ * the world.
+ */
+ if (OidIsValid(OIDOldToast))
+ {
+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);
+ if (!OidIsValid(toast_chunkid_typid))
+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);
+ }
+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -353,7 +371,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;
+ if (toast_chunkid_typid == OID8OID)
+ opclassIds[0] = OID8_BTREE_OPS_OID;
+ else
+ opclassIds[0] = OID_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 534fe338b386..dd4b6e8f7f45 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5036,6 +5036,8 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
TupleDesc desc = RelationGetDescr(relation);
Oid8 chunk_id;
int32 chunk_seq;
+ Oid valueid_type;
+ Datum valueid_datum;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
@@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));
+ valueid_type = TupleDescAttr(desc, 0)->atttypid;
+ valueid_datum = fastgetattr(newtup, 1, desc, &isnull);
+ if (valueid_type == OID8OID)
+ chunk_id = DatumGetObjectId8(valueid_datum);
+ else
+ chunk_id = DatumGetObjectId(valueid_datum);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out
index 9b4c8587aa49..95e4528f7793 100644
--- a/src/test/regress/expected/reloptions.out
+++ b/src/test/regress/expected/reloptions.out
@@ -199,7 +199,7 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
ERROR: unrecognized parameter "toast_value_type"
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
ERROR: invalid value for enum option "toast_value_type": int8
-DETAIL: Valid values are "oid".
+DETAIL: Valid values are "oid" and "oid8".
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
reloptions
@@ -207,6 +207,19 @@ SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
{toast_value_type=oid}
(1 row)
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ chunk_id_type
+---------------
+ oid
+(1 row)
+
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
DROP TABLE reloptions_test2;
-- relkinds not supported.
CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql
index 503d15d16e51..80939ee8b5bd 100644
--- a/src/test/regress/sql/reloptions.sql
+++ b/src/test/regress/sql/reloptions.sql
@@ -117,6 +117,14 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid');
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');
CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid');
SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;
+-- The option is only consulted when the TOAST relation is created, so
+-- changing it afterwards leaves chunk_id alone.
+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8');
+SELECT a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = 'reloptions_test2'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+ALTER TABLE reloptions_test2 RESET (toast_value_type);
DROP TABLE reloptions_test2;
-- relkinds not supported.
CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 79622c1a23d6..6f9a87ef449f 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1663,7 +1663,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<para>
Specifies the attribute type of <literal>chunk_id</literal> to use when
creating a <acronym>TOAST</acronym> relation for this table. The
- default is <literal>oid</literal>.
+ default is <literal>oid</literal>; <literal>oid8</literal> assigns
+ <type>oid8</type> instead, which allows a larger number of distinct
+ <acronym>TOAST</acronym>ed values at the price of four extra bytes per
+ out-of-line pointer.
This parameter cannot be set for <acronym>TOAST</acronym> tables.
</para>
<para>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 90aae3defcbc..21bd67c11ea0 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID
-identifying the particular <acronym>TOAST</acronym>ed value),
+<acronym>TOAST</acronym> table has the columns
+<structfield>chunk_id</structfield> (an OID or an OID8 identifying
+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value
+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 33b00fa4dc5c..56167ecfe1d0 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -28,7 +28,6 @@
#include "storage/procarray.h"
#include "storage/read_stream.h"
#include "utils/builtins.h"
-#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/tuplestore.h"
@@ -1876,8 +1875,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;
+
+ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1885,10 +1888,8 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,
- (AttrNumber) 1,
- BTEqualStrategyNumber, F_OIDEQ,
- ObjectIdGetDatum(ta->toast_pointer.va_valueid));
+ toast_valueid_scankey_init(&toastkey, toast_typid,
+ ta->toast_pointer.va_valueid);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v18-0002-Add-battery-of-tests-related-oid8.patch (text/plain, 38.3K) download
From 328b18a81482b1b080869e95559ddb9893042dd5 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 14 Sep 2026 13:17:51 +0900
Subject: [PATCH v18 2/5] Add battery of tests related oid8
The tests added by this commit cover a large ground in terms of the
introduction of oid8 as external TOAST pointer, with the following areas
covered, as found during development of the feature:
- Tables with oid8 TOAST tables in general, under strings.sql.
- Relation rewrites do not affect the TOAST relation after initial
creation.
- oid8 TOAST with pg_column_compression().
- test_decoding, for external and extended TOAST data with oid8
pointers.
- amcheck, with oid8 TOAST relations, external and extended formats with
heap verified.
- UPDATE with out-of-line datum that belongs to another TOAST table.
This tracks a bug in toast_tuple_init(), where external vartags could
be mixed once multiple types are introduced. That's something that
absan would trigger.
These tests are added before the commit that has adds the new external
vartag, just because it is what it is.
---
src/test/regress/expected/compression.out | 41 +++
src/test/regress/expected/strings.out | 318 +++++++++++++++++++---
src/test/regress/sql/compression.sql | 22 ++
src/test/regress/sql/strings.sql | 193 ++++++++++---
contrib/amcheck/expected/check_heap.out | 40 +++
contrib/amcheck/sql/check_heap.sql | 27 ++
contrib/test_decoding/expected/toast.out | 39 +++
contrib/test_decoding/sql/toast.sql | 20 ++
8 files changed, 625 insertions(+), 75 deletions(-)
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 09f198149aa4..e9128193515a 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
(1 row)
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+ uncompressed
+--------------
+ t
+(1 row)
+
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+ uncompressed
+--------------
+ t
+(1 row)
+
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+DROP TABLE toastcomp_oid, toastcomp_oid8;
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index 313c50394650..95328b304bf5 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;
+DROP TABLE toasttest_oid;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ f
+(1 row)
+
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+ is_empty
+----------
+ t
+(1 row)
+
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+ substr
+--------
+ 123
+ 123
+ 123
+ 123
+(4 rows)
+
+SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+ERROR: negative substring length not allowed
+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2105,7 +2198,158 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+ substr
+--------
+ 567890
+ 567890
+ 567890
+ 567890
+(4 rows)
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+REPACK toasttest_oid;
+REPACK toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ relname | atttypid
+----------------+----------
+ toasttest_oid | oid
+ toasttest_oid8 | oid8
+(2 rows)
+
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+ length | substr
+--------+--------
+ 100000 | aaa
+(1 row)
+
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+ count
+-------
+ 1
+(1 row)
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index ce5ea37a660c..229143f06ddd 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2;
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8
+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;
+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;
+-- out-of-line and compressed.
+TRUNCATE toastcomp_oid;
+TRUNCATE toastcomp_oid8;
+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);
+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);
+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));
+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));
+SELECT pg_column_compression(f1) FROM toastcomp_oid;
+SELECT pg_column_compression(f1) FROM toastcomp_oid8;
+DROP TABLE toastcomp_oid, toastcomp_oid8;
+
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql
index 38946e8954df..4babfc883a9b 100644
--- a/src/test/regress/sql/strings.sql
+++ b/src/test/regress/sql/strings.sql
@@ -563,89 +563,206 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values
+-- Test substr with toasted text values, for all types of TOAST relations
+-- supported.
--
-CREATE TABLE toasttest(f1 text);
+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(repeat('1234567890',10000));
-insert into toasttest values(repeat('1234567890',10000));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(repeat('1234567890',10000));
+insert into toasttest_oid values(repeat('1234567890',10000));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(repeat('1234567890',10000));
+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-TRUNCATE TABLE toasttest;
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+-- TRUNCATE cases for TOAST relations with OID values.
+TRUNCATE TABLE toasttest_oid;
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
-
-TRUNCATE TABLE toasttest;
-ALTER TABLE toasttest set (toast_tuple_target = 4080);
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
-INSERT INTO toasttest values (repeat('1234567890',300));
+ FROM pg_class where relname = 'toasttest_oid';
+TRUNCATE TABLE toasttest_oid;
+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';
+ FROM pg_class where relname = 'toasttest_oid';
+DROP TABLE toasttest_oid;
-DROP TABLE toasttest;
+-- TRUNCATE cases for TOAST relation with int8 values.
+TRUNCATE TABLE toasttest_oid8;
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect >0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+TRUNCATE TABLE toasttest_oid8;
+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));
+-- expect 0 blocks
+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
+ FROM pg_class where relname = 'toasttest_oid8';
+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values
+-- test substr with toasted bytea values, for all types of TOAST relations
+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);
+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');
+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid alter column f1 set storage external;
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));
+alter table toasttest_oid8 alter column f1 set storage external;
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;
+SELECT substr(f1, -1, 5) from toasttest_oid;
+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;
+SELECT substr(f1, 5, -1) from toasttest_oid;
+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
-DROP TABLE toasttest;
+-- A relation rewrite leaves the TOAST value attributes unchanged.
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+-- Check that data slices are still accessible.
+SELECT substr(f1, 99995) from toasttest_oid;
+SELECT substr(f1, 99995) from toasttest_oid8;
+SELECT substr(f1, 99995, 10) from toasttest_oid;
+SELECT substr(f1, 99995, 10) from toasttest_oid8;
+
+-- ALTER TABLE after TOAST table creation does not affect relation rewrite.
+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);
+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);
+VACUUM FULL toasttest_oid;
+VACUUM FULL toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+REPACK toasttest_oid;
+REPACK toasttest_oid8;
+SELECT c1.relname, a.atttypid::regtype
+ FROM pg_attribute AS a,
+ pg_class AS c1,
+ pg_class AS c2
+ WHERE
+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND
+ c1.reltoastrelid = c2.oid AND
+ a.attrelid = c2.oid AND
+ a.attname = 'chunk_id'
+ ORDER BY c1.relname COLLATE "C";
+ALTER TABLE toasttest_oid RESET (toast_value_type);
+ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+
+-- Reset column storage to its default
+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+
+-- UPDATE with out-of-line datum that belongs to another TOAST table.
+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');
+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');
+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;
+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;
+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class
+ WHERE oid = 'toastupd_oid'::regclass \gset
+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class
+ WHERE oid = 'toastupd_oid8'::regclass \gset
+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));
+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));
+-- old value is an oid8 pointer, new value an oid pointer.
+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;
+-- replaced value must be gone, leaving a single value behind.
+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;
+-- reverse: old value is an oid pointer, new value an oid8 pointer.
+TRUNCATE toastupd_oid;
+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));
+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;
+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;
+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;
+
+DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index 979e5e84e723..8686fba712d0 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -199,6 +199,44 @@ SELECT * FROM verify_heapam('test_partition',
-------+--------+--------+-----
(0 rows)
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+ relname | chunk_id_type
+-----------------+---------------
+ test_toast_oid | oid
+ test_toast_oid8 | oid8
+(2 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+ blkno | offnum | attnum | msg
+-------+--------+--------+-----
+(0 rows)
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -232,6 +270,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 1745bae634e5..eb46fe710351 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -112,6 +112,31 @@ SELECT * FROM verify_heapam('test_partition',
startblock := NULL,
endblock := NULL);
+-- Check TOAST relations of both chunk_id: oid and oid8
+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');
+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');
+-- Uncompressed out-of-line values
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
+-- Compressed out-of-line values.
+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO test_toast_oid (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+INSERT INTO test_toast_oid8 (a, b)
+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+SELECT c.relname, a.atttypid::regtype AS chunk_id_type
+ FROM pg_class AS c, pg_attribute AS a
+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'
+ ORDER BY c.relname COLLATE "C";
+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);
+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);
+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -139,6 +164,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
endblock := NULL);
-- cleanup
+DROP TABLE test_toast_oid;
+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.out
index a757e7dc8d54..8bef6cc32af4 100644
--- a/contrib/test_decoding/expected/toast.out
+++ b/contrib/test_decoding/expected/toast.out
@@ -382,6 +382,45 @@ SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot',
COMMIT
(4 rows)
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ shortened | len
+------------------------------------------------------------------------------------+--------
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:1 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: INSERT: id[integer]:2 data[text]:..2345678901234567890' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: old-key: id[integer]:1 ne..nchanged-toast-datum | 116
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: UPDATE: id[integer]:2 data[text]:..bcdefghijabcdefghij' | 200062
+ COMMIT | 6
+ BEGIN | 5
+ table public.toasted_oid8: DELETE: id[integer]:11 | 49
+ table public.toasted_oid8: DELETE: id[integer]:2 | 48
+ COMMIT | 6
+(16 rows)
+
+DROP TABLE toasted_oid8;
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sql
index d1c560a174d6..8a2c49a5c9a7 100644
--- a/contrib/test_decoding/sql/toast.sql
+++ b/contrib/test_decoding/sql/toast.sql
@@ -324,4 +324,24 @@ INSERT INTO tbl2 VALUES(1);
commit;
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- Test decoding of TOAST values with oid8
+CREATE TABLE toasted_oid8 (id serial primary key, data text)
+ WITH (toast_value_type = 'oid8');
+-- uncompressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- compressed external toast data
+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;
+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));
+-- update without changing the toasted column, reported as unchanged
+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;
+-- update changing the toasted column
+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;
+DELETE FROM toasted_oid8;
+-- Check that the values are reassembled in full.
+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,
+ length(data) AS len
+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+DROP TABLE toasted_oid8;
+
SELECT pg_drop_replication_slot('regression_slot');
--
2.55.0
v18-0003-Add-support-for-TOAST-pointers-as-oid8.patch (text/plain, 46.7K) download
From 5b63e12d4d95311b4d4816586951c31199ad0a2a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 11 Sep 2026 15:57:31 +0900
Subject: [PATCH v18 3/5] Add support for TOAST pointers as oid8
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 49 ++++
src/include/access/heaptoast.h | 22 +-
src/include/utils/rel.h | 9 +
src/include/utils/relcache.h | 1 +
src/include/varatt.h | 78 ++++--
src/backend/access/common/detoast.c | 75 +++---
src/backend/access/common/toast_compression.c | 8 +-
src/backend/access/common/toast_internals.c | 241 ++++++++++--------
src/backend/access/heap/heaptoast.c | 2 +-
src/backend/access/table/toast_helper.c | 22 +-
.../replication/logical/reorderbuffer.c | 15 +-
src/backend/utils/adt/varlena.c | 8 +-
src/backend/utils/cache/relcache.c | 37 +++
doc/src/sgml/storage.sgml | 12 +-
contrib/amcheck/verify_heapam.c | 58 +++--
src/tools/pgindent/typedefs.list | 2 +
16 files changed, 430 insertions(+), 209 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 2b1428b1e81c..e9d8076f77cd 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -12,6 +12,8 @@
#ifndef DETOAST_H
#define DETOAST_H
+#include "varatt.h"
+
/*
* Macro to fetch the possibly-unaligned contents of an EXTERNAL datum
* into a local "varatt_external_oid" toast pointer. This should be
@@ -30,9 +32,56 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */
+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))
+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
+/*
+ * Decoded contents of an on-disk TOAST pointer, independent of whether the
+ * value ID type.
+ */
+typedef struct toast_external_data
+{
+ vartag_external tag; /* VARTAG_ONDISK_* */
+ int32 rawsize; /* original data size (includes header) */
+ uint32 extinfo; /* saved size + compression method */
+ Oid8 valueid; /* value ID (can be widened from Oid) */
+ Oid toastrelid; /* OID of the TOAST table containing it */
+} toast_external_data;
+
+/*
+ * Decode an on-disk TOAST pointer into a toast_external_data.
+ */
+static inline void
+toast_external_info_get(const struct varlena *attr, toast_external_data *toast_ext_data)
+{
+ Assert(VARATT_IS_EXTERNAL_ONDISK(attr));
+
+ toast_ext_data->tag = VARTAG_EXTERNAL(attr);
+ if (toast_ext_data->tag == VARTAG_ONDISK_OID8)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_ext_data->rawsize = toast_pointer.va_rawsize;
+ toast_ext_data->extinfo = toast_pointer.va_extinfo;
+ toast_ext_data->valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);
+ toast_ext_data->toastrelid = toast_pointer.va_toastrelid;
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_ext_data->rawsize = toast_pointer.va_rawsize;
+ toast_ext_data->extinfo = toast_pointer.va_extinfo;
+ toast_ext_data->valueid = toast_pointer.va_valueid;
+ toast_ext_data->toastrelid = toast_pointer.va_toastrelid;
+ }
+}
+
/* ----------
* detoast_external_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index aaae6e0fef69..4c3fda8a5f9a 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*
- * be small enough that the completed toast-table tuple (including the
- * ID and sequence fields and all overhead) will fit on a page.
+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE
+ * data bytes, depending on the chunk_id type of the TOAST table. These
+ * numbers *must* be small enough that the completed toast-table tuple
+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.
+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,19 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \
+ (EXTERN_TUPLE_MAX_SIZE - \
+ MAXALIGN(SizeofHeapTupleHeader) - \
+ sizeof(Oid8) - \
+ sizeof(int32) - \
+ VARHDRSZ)
+
+/*
+ * Select the chunk size for a TOAST table from its value type.
+ */
+#define TOAST_MAX_CHUNK_SIZE(toast_typid) \
+ ((toast_typid) == OID8OID ? TOAST_OID8_MAX_CHUNK_SIZE : TOAST_OID_MAX_CHUNK_SIZE)
+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index d61432de705e..916b67f5504f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -250,6 +250,15 @@ typedef struct RelationData
*/
Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */
+ /*
+ * Type OID of the "chunk_id" column of this relation's TOAST table, i.e.
+ * OIDOID or OID8OID.
+ *
+ * This data is filled on demand by RelationGetToastChunkIdType(), not
+ * at relcache build time, so as to save in syscache lookups.
+ */
+ Oid rd_toastchunkidtype;
+
bool pgstat_enabled; /* should relation stats be counted */
/* use "struct" here to avoid needing to include pgstat.h: */
struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 89c27aa1529f..e17c84908332 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok);
extern Oid RelationGetReplicaIndex(Relation relation);
+extern Oid RelationGetToastChunkIdType(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
extern List *RelationGetDummyIndexExpressions(Relation relation);
extern List *RelationGetIndexPredicate(Relation relation);
diff --git a/src/include/varatt.h b/src/include/varatt.h
index 89b47db08332..77041f7673a9 100644
--- a/src/include/varatt.h
+++ b/src/include/varatt.h
@@ -42,6 +42,44 @@ StaticAssertDecl((sizeof(int32) + sizeof(uint32) + 2 * sizeof(Oid)) ==
sizeof(varatt_external_oid),
"varatt_external_oid should have no padding");
+/*
+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use
+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid
+ * except for the value ID, which is 8 bytes wide. The value ID is split
+ * into two uint32 to force alignment.
+ *
+ * This struct must not contain any padding, because we sometimes compare
+ * these pointers using memcmp.
+ */
+typedef struct varatt_external_oid8
+{
+ int32 va_rawsize; /* Original data size (includes header) */
+ uint32 va_extinfo; /* External saved size (without header) and
+ * compression method */
+ uint32 va_valueid_lo; /* Low 32 bits of value ID */
+ uint32 va_valueid_hi; /* High 32 bits of value ID */
+ Oid va_toastrelid; /* RelID of TOAST table containing it */
+} varatt_external_oid8;
+
+StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==
+ sizeof(varatt_external_oid8),
+ "varatt_external_oid8 should have no padding");
+
+/* Get/set the 64-bit value ID from the lo/hi halves */
+static inline Oid8
+VARATT_EXTERNAL_OID8_GET_VALUEID(varatt_external_oid8 toast_pointer)
+{
+ return ((Oid8) toast_pointer.va_valueid_lo) |
+ (((Oid8) toast_pointer.va_valueid_hi) << 32);
+}
+
+static inline void
+VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id)
+{
+ toast_pointer->va_valueid_lo = (uint32) id;
+ toast_pointer->va_valueid_hi = (uint32) (id >> 32);
+}
+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -91,6 +129,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -112,6 +151,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)
+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -365,7 +406,12 @@ VARATT_IS_EXTERNAL(const void *PTR)
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
{
- return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
+ vartag_external tag;
+
+ if (!VARATT_IS_EXTERNAL(PTR))
+ return false;
+ tag = VARTAG_EXTERNAL(PTR);
+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);
}
/* Is varlena datum an indirect pointer? */
@@ -508,43 +554,31 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR)
}
/*
- * Same for external Datums; but note argument is a struct
- * varatt_external_oid.
+ * Same for external Datums, saved into an va_extinfo.
*/
static inline Size
-VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_GET_EXTSIZE(uint32 extinfo)
{
- return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;
+ return extinfo & VARLENA_EXTSIZE_MASK;
}
static inline uint32
-VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_GET_COMPRESS_METHOD(uint32 extinfo)
{
- return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
+ return extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */
-/* This has to remain a macro; beware multiple evaluations! */
-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \
- do { \
- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \
- (cm) == TOAST_LZ4_COMPRESSION_ID); \
- ((toast_pointer).va_extinfo = \
- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \
- } while (0)
-
/*
- * Testing whether an externally-stored value is compressed now requires
- * comparing size stored in va_extinfo (the actual length of the external data)
+ * Testing whether an externally-stored value is compressed requires comparing
+ * the saved size stored in va_extinfo (the actual length of the external data)
* to rawsize (the original uncompressed datum's size). The latter includes
* VARHDRSZ overhead, the former doesn't. We never use compression unless it
* actually saves space, so we expect either equality or less-than.
*/
static inline bool
-VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
+VARATT_EXTINFO_IS_COMPRESSED(uint32 extinfo, int32 rawsize)
{
- return VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer) <
- (Size) (toast_pointer.va_rawsize - VARHDRSZ);
+ return VARATT_EXTINFO_GET_EXTSIZE(extinfo) < (Size) (rawsize - VARHDRSZ);
}
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a50aea7b6b12..7e5863262b3d 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -225,12 +225,18 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
+ int32 extsize;
+ uint32 compress_method;
+ bool is_compressed;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
+ extsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ compress_method = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
+ is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize);
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +246,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ int32 max_size = extsize;
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,14 +257,9 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==
- TOAST_PGLZ_COMPRESSION_ID)
+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*
- * Fetch enough compressed slices (compressed marker will get set
- * automatically).
- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,20 +345,22 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toast_external_info_get(attr, &toast_ext_data);
+ attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ toastrelid = toast_ext_data.toastrelid;
+ valueid = toast_ext_data.valueid;
result = (varlena *) palloc(attrsize + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);
else
SET_VARSIZE(result, attrsize + VARHDRSZ);
@@ -369,10 +372,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +401,27 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
int32 attrsize;
+ Oid toastrelid;
+ Oid8 valueid;
+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
+ attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
+ toastrelid = toast_ext_data.toastrelid;
+ valueid = toast_ext_data.valueid;
+ is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize);
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);
-
- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +434,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)
+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +447,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +456,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +557,10 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = toast_pointer.va_rawsize;
+ toast_external_info_get(attr, &toast_ext_data);
+ result = toast_ext_data.rawsize;
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +617,10 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);
+ toast_external_info_get(attr, &toast_ext_data);
+ result = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c
index 3b1bd6519261..09cdf880e4ff 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -262,12 +262,12 @@ toast_get_compression_id(varlena *attr)
*/
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
- cmid = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
+ cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
}
else if (VARATT_IS_COMPRESSED(attr))
cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index de4b17fa8164..2b35165d48f8 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -25,7 +25,6 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -104,6 +103,57 @@ toast_compress_datum(Datum value, char cmethod)
}
}
+/* ----------
+ * toast_preserve_valueid -
+ *
+ * During a table rewrite that preserves rd_toastoid, we want to preserve
+ * toast value IDs too. If the datum previously had an external value from
+ * that same toast table, return its value ID so the caller can re-use it,
+ * otherwise return InvalidOid8.
+ *
+ * This works for both Oid and Oid8 value IDs, as the value ID is decoded
+ * independently of the pointer's vartag.
+ *
+ * There is a corner case here: the table rewrite might have to copy both
+ * live and recently-dead versions of a row, and those versions could easily
+ * reference the same toast value. When we copy the second or later version
+ * of such a row, preserving the value ID means we select one that's already
+ * in the new toast table. We detect that and set *data_todo to 0 so the
+ * caller falls through without writing the data again.
+ *
+ * While annoying and ugly-looking, this is a good thing because it ensures
+ * that we wind up with only one copy of the toast value when there is only
+ * one copy in the old toast table. Before we detected this case, we'd have
+ * made multiple copies, wasting space; and what's worse, the copies
+ * belonging to already-deleted heap tuples would not be reclaimed by VACUUM.
+ * ----------
+ */
+static Oid8
+toast_preserve_valueid(Relation toastrel, Oid toastoid,
+ varlena *oldexternal, int32 *data_todo)
+{
+ toast_external_data old;
+
+ if (!OidIsValid(toastoid) || oldexternal == NULL)
+ return InvalidOid8;
+
+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
+ toast_external_info_get(oldexternal, &old);
+
+ /*
+ * Only re-use the value ID if the old pointer came from this same toast
+ * table. Since that is the very same relation, its chunk_id type
+ * necessarily matches, so no separate vartag check is needed.
+ */
+ if (old.toastrelid != toastoid)
+ return InvalidOid8;
+
+ if (toastrel_valueid_exists(toastrel, old.valueid))
+ *data_todo = 0;
+
+ return old.valueid;
+}
+
/* ----------
* toast_save_datum -
*
@@ -125,14 +175,20 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
- Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ Oid toast_typid = RelationGetToastChunkIdType(rel);
+ int32 max_chunk_size;
+
+ /* Fields that will be assembled into the TOAST pointer at the end */
+ int32 va_rawsize;
+ uint32 va_extinfo;
+ Oid8 va_valueid;
+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
@@ -164,28 +220,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */
+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);
+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,
- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));
+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||
+ cmid == TOAST_LZ4_COMPRESSION_ID);
+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);
+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));
+ Assert(VARATT_EXTINFO_IS_COMPRESSED(va_extinfo, va_rawsize));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);
- toast_pointer.va_extinfo = data_todo;
+ va_rawsize = VARSIZE(dval);
+ va_extinfo = data_todo;
}
/*
@@ -197,101 +257,49 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;
+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it
- * for OID or OID8 TOAST relations.
- *
- * Normally we just choose an unused value within the toast table. But
- * during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value IDs too. So, if
- * rd_toastoid is set and we had a prior external value from that same
- * toast table, re-use its value ID. If we didn't have a prior external
- * value (which is a corner case, but possible if the table's attstorage
- * options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST
- * table uses 8-byte value IDs, we should not really care much about
- * that.
+ * Choose the value ID for this toast value. During a table rewrite that
+ * preserves rd_toastoid we re-use the prior value ID if we can (see
+ * toast_preserve_valueid); otherwise we pick a fresh one. For Oid value
+ * IDs the fresh one must not conflict with old or new toast values; for
+ * Oid8 the ID space is large enough that conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))
+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */
- if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
- else
+ va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid,
+ oldexternal, &data_todo);
+ if (va_valueid == InvalidOid8)
{
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
+ /* The Oid8 space is large enough that we need not check conflicts */
+ va_valueid = GetNewObjectId8();
}
}
else
{
- /* rewrite case: check to see if value was in old toast table */
- toast_pointer.va_valueid = InvalidOid;
- if (oldexternal != NULL)
- {
- varatt_external_oid old_toast_pointer;
-
- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);
- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)
- {
- /* This value came from the old toast table; reuse its OID */
- toast_pointer.va_valueid = old_toast_pointer.va_valueid;
-
- /*
- * There is a corner case here: the table rewrite might have
- * to copy both live and recently-dead versions of a row, and
- * those versions could easily reference the same toast value.
- * When we copy the second or later version of such a row,
- * reusing the OID will mean we select an OID that's already
- * in the new toast table. Check for that, and if so, just
- * fall through without writing the data again.
- *
- * While annoying and ugly-looking, this is a good thing
- * because it ensures that we wind up with only one copy of
- * the toast value when there is only one copy in the old
- * toast table. Before we detected this case, we'd have made
- * multiple copies, wasting space; and what's worse, the
- * copies belonging to already-deleted heap tuples would not
- * be reclaimed by VACUUM.
- */
- if (toastrel_valueid_exists(toastrel,
- toast_pointer.va_valueid))
- {
- /* Match, so short-circuit the data storage loop below */
- data_todo = 0;
- }
- }
- }
- if (toast_pointer.va_valueid == InvalidOid)
+ va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid,
+ oldexternal, &data_todo);
+ if (va_valueid == InvalidOid)
{
/*
- * new value; must choose a value that doesn't conflict in either
- * old or new toast table.
+ * Choose an unused OID. During a rewrite we must also avoid IDs
+ * that are still present in the old toast table.
*/
- if (toast_typid == OID8OID)
- toast_pointer.va_valueid = GetNewObjectId8();
- else
- {
- do
- {
- toast_pointer.va_valueid =
- GetNewOidWithIndex(toastrel,
- RelationGetRelid(toastidxs[validIndex]),
- (AttrNumber) 1);
- } while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));
- }
+ do
+ va_valueid =
+ GetNewOidWithIndex(toastrel,
+ RelationGetRelid(toastidxs[validIndex]),
+ (AttrNumber) 1);
+ while (OidIsValid(rel->rd_toastoid) &&
+ toastid_valueid_exists(rel->rd_toastoid, va_valueid));
}
}
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
+
/*
* Split up the item into chunks
*/
@@ -313,15 +321,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);
+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
if (toast_typid == OID8OID)
- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
+ t_values[0] = ObjectId8GetDatum(va_valueid);
else
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);
+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -375,11 +383,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return
+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ if (toast_typid == OID8OID)
+ {
+ varatt_external_oid8 toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ VARATT_EXTERNAL_OID8_SET_VALUEID(&toast_pointer, va_valueid);
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
+ else
+ {
+ varatt_external_oid toast_pointer;
+
+ toast_pointer.va_rawsize = va_rawsize;
+ toast_pointer.va_extinfo = va_extinfo;
+ toast_pointer.va_valueid = (Oid) va_valueid;
+ toast_pointer.va_toastrelid = va_toastrelid;
+
+ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);
+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);
+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));
+ }
return PointerGetDatum(result);
}
@@ -414,7 +445,7 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -426,15 +457,17 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ /*
+ * Decode the pointer to get the toast relation OID and value ID. The
+ * vartag tells us everything we need - no TOAST table schema lookup
+ * required.
+ */
+ toast_external_info_get(attr, &toast_ext_data);
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
-
- /* Fetch valid relation used for process */
+ toastrel = table_open(toast_ext_data.toastrelid, RowExclusiveLock);
validIndex = toast_open_indexes(toastrel,
RowExclusiveLock,
&toastidxs,
@@ -445,7 +478,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
*/
toast_valueid_scankey_init(&toastkey,
TupleDescAttr(toastrel->rd_att, 0)->atttypid,
- toast_pointer.va_valueid);
+ toast_ext_data.valueid);
/*
* Find all the chunks. (We don't actually care whether we see them in
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 81d5085ca398..f8a54ff2582e 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -649,7 +649,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&num_indexes);
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 6f2d691cc681..2613d9dc095b 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc)
/*
* If the old value is stored on disk, check if it has changed so
* we have to delete it later.
+ *
+ * Note that TOAST pointers could have different vartags, for oid
+ * or oid8, and these can have different sizes.
*/
if (att->attlen == -1 && !ttc->ttc_oldisnull[i] &&
VARATT_IS_EXTERNAL_ONDISK(old_value))
{
if (ttc->ttc_isnull[i] ||
!VARATT_IS_EXTERNAL_ONDISK(new_value) ||
+ VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) ||
memcmp(old_value, new_value,
VARSIZE_EXTERNAL(old_value)) != 0)
{
@@ -171,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);
- * if not, no benefit is to be expected by compressing it.
+ * The column must be larger than the TOAST pointer that would replace it;
+ * if not, no benefit is to be expected by compressing it. Note that this
+ * choice depends on the TOAST value type, oid or oid8.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -184,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+ int32 biggest_size;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /*
+ * Size of the TOAST pointer this relation would use. A relation without
+ * a TOAST table cannot have any of its attributes moved out-of-line, but
+ * it can still have some of them compressed. Fall back to the oid size
+ * in that case.
+ */
+ if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID)
+ biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE);
+ else
+ biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);
+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dd4b6e8f7f45..69a479cd31da 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5175,7 +5175,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *varlena_pointer;
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
@@ -5198,11 +5198,11 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena_pointer = (varlena *) DatumGetPointer(attrs[natt]);
/* no need to do anything if the tuple isn't external */
- if (!VARATT_IS_EXTERNAL(varlena_pointer))
+ if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
- toast_valueid = toast_pointer.va_valueid;
+ toast_external_info_get(varlena_pointer, &toast_ext_data);
+ toast_valueid = toast_ext_data.valueid;
/*
* Check whether the toast tuple changed, replace if so.
@@ -5220,7 +5220,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);
+ reconstructed = palloc0(toast_ext_data.rawsize);
ent->reconstructed = reconstructed;
@@ -5245,10 +5245,11 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));
+
+ Assert(data_done == VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo));
/* make sure its marked as compressed or not */
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);
else
SET_VARSIZE(reconstructed, data_done + VARHDRSZ);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 4533dac11a6d..f8c16bd4c871 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -4261,7 +4261,8 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external_oid toast_pointer;
+ toast_external_data toast_ext_data;
+ Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4288,9 +4289,10 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_external_info_get(attr, &toast_ext_data);
+ result = toast_ext_data.valueid;
- PG_RETURN_OID8(toast_pointer.va_valueid);
+ PG_RETURN_OID8(result);
}
/*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 541f30f0972c..2555de937067 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1205,6 +1205,9 @@ retry:
relation->rd_fkeylist = NIL;
relation->rd_fkeyvalid = false;
+ /* TOAST type data is not loaded till asked for */
+ relation->rd_toastchunkidtype = InvalidOid;
+
/* partitioning data is not loaded till asked for */
relation->rd_partkey = NULL;
relation->rd_partkeycxt = NULL;
@@ -5100,6 +5103,39 @@ RelationGetReplicaIndex(Relation relation)
return relation->rd_replidindex;
}
+/*
+ * RelationGetToastChunkIdType -- get the type of the relation's TOAST
+ * table "chunk_id" column
+ *
+ * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST
+ * table.
+ */
+Oid
+RelationGetToastChunkIdType(Relation relation)
+{
+ Oid toastrelid = relation->rd_rel->reltoastrelid;
+ Oid typid;
+
+ /* Quick exit if we already computed the value */
+ if (OidIsValid(relation->rd_toastchunkidtype))
+ return relation->rd_toastchunkidtype;
+
+ /* Nothing to report without a TOAST table */
+ if (!OidIsValid(toastrelid))
+ return InvalidOid;
+
+ typid = get_atttype(toastrelid, 1);
+ if (!OidIsValid(typid))
+ elog(ERROR, "cache lookup failed for TOAST relation %u",
+ toastrelid);
+ if (typid != OIDOID && typid != OID8OID)
+ elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u",
+ typid, toastrelid);
+
+ relation->rd_toastchunkidtype = typid;
+ return typid;
+}
+
/*
* RelationGetIndexExpressions -- get the index expressions for an index
*
@@ -6545,6 +6581,7 @@ load_relcache_init_file(bool shared)
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
rel->rd_droppedSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_toastchunkidtype = InvalidOid;
rel->pgstat_info = NULL;
/*
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 21bd67c11ea0..83de016eaa58 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -417,8 +417,10 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen
-so that four chunk rows will fit on a page, making it about 2000 bytes).
+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or
+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the
+<structfield>chunk_id</structfield> type (by default these values are chosen
+so that four chunk rows will fit on a page, making them about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
<acronym>TOAST</acronym> table has the columns
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18
-bytes regardless of the actual size of the represented value.
+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18
+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes
+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the
+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 56167ecfe1d0..bb97912a7746 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -74,7 +74,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;
+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */
+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */
+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1565,9 +1567,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 max_chunk_size;
Oid8 toast_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ toast_valueid = ta->va_valueid;
+
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1671,8 +1676,11 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;
+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;
+ vartag_external va_tag_value;
+ toast_external_data toast_ext_data;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1731,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)
+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1775,27 +1783,28 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
- /*
- * Must copy attr into toast_pointer for alignment considerations
- */
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- toast_pointer_valueid = toast_pointer.va_valueid;
+ /* Must copy attr into a decoded pointer for alignment considerations */
+ toast_external_info_get(attr, &toast_ext_data);
+ va_tag_value = toast_ext_data.tag;
+ toast_pointer_valueid = toast_ext_data.valueid;
+ va_rawsize = toast_ext_data.rawsize;
+ va_extinfo = toast_ext_data.extinfo;
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,
+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))
+ if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);
+ cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo);
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1849,7 +1858,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);
+ /* The pointer has already been decoded above, just reuse it */
+ ta->tag = va_tag_value;
+ ta->va_valueid = toast_pointer_valueid;
+ ta->va_extinfo = va_extinfo;
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1879,17 +1891,19 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
Oid8 toast_valueid;
Oid toast_typid;
- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
-
- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
- last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->va_valueid;
+ extsize = VARATT_EXTINFO_GET_EXTSIZE(ta->va_extinfo);
+ toast_typid = (ta->tag == VARTAG_ONDISK_OID8) ? OID8OID : OIDOID;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
+ max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid);
toast_valueid_scankey_init(&toastkey, toast_typid,
- ta->toast_pointer.va_valueid);
+ toast_valueid);
+
+ last_chunk_seq = (extsize - 1) / max_chunk_size;
+
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1909,8 +1923,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;
-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cdf5a7d0b11b..8c38f8c275dd 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4360,6 +4360,7 @@ timeout_params
timerCA
tlist_vinfo
toast_compress_header
+toast_external_data
tokenize_error_callback_arg
transferMode
transfer_thread_arg
@@ -4416,6 +4417,7 @@ vacuumingOptions
validate_string_relopt
varatt_expanded
varatt_external_oid
+varatt_external_oid8
varatt_indirect
varattrib_1b
varattrib_1b_e
--
2.55.0
v18-0004-Decode-an-oid8-TOAST-value-with-an-ID-past-2-32.patch (text/plain, 2.5K) download
From 05b107faaf655209c9627d5e44c2941f74366b1d Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 14 Sep 2026 13:36:12 +0900
Subject: [PATCH v18 4/5] Decode an oid8 TOAST value with an ID past 2^32
The test_decoding tests run with a young OID counter, so the value IDs
they see fit in 4 bytes and a chunk_id read as an Oid still matches.
Move the counter past 2^32 in the logical decoding TAP test and decode
one oid8 value there.
Author: Rui Zhao <zhaorui126@gmail.com>
---
src/test/recovery/t/006_logical_decoding.pl | 33 +++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 97d11f98b59a..57d3cf91ee21 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -20,6 +20,13 @@ $node_primary->append_conf(
'postgresql.conf', qq(
wal_level = logical
));
+
+# Move the OID counter past 2^32, so that the oid8 TOAST values decoded at
+# the end of this test have IDs that do not fit in 4 bytes.
+my $next_oid = '4295067296'; # 2^32 + 100000
+command_ok(
+ [ 'pg_resetwal', '--next-oid' => $next_oid, $node_primary->data_dir ],
+ 'set an 8-byte OID counter');
$node_primary->start;
$node_primary->safe_psql('postgres',
@@ -275,6 +282,32 @@ is( $node_primary->safe_psql(
qq(Check that reset timestamp is later after resetting stats for slot '$stats_test_slot1' again.)
);
+# Decode an oid8 TOAST value whose ID is past 2^32. The reorder buffer
+# collects the chunks by chunk_id, and has to read that column with the
+# TOAST table's type, or the value is reported as unchanged.
+$node_primary->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE toasted_oid8 (id int PRIMARY KEY, data text)
+ WITH (toast_value_type = 'oid8');
+ ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;
+ SELECT pg_create_logical_replication_slot('oid8_slot', 'test_decoding');
+ INSERT INTO toasted_oid8 VALUES (1, repeat('1234567890', 2000));
+]);
+is( $node_primary->safe_psql(
+ 'postgres',
+ "SELECT pg_column_toast_chunk_id(data) > '$next_oid'::oid8 FROM toasted_oid8"
+ ),
+ 't',
+ 'oid8 TOAST value has an ID past 2^32');
+$result = $node_primary->safe_psql(
+ 'postgres',
+ "SELECT data FROM pg_logical_slot_get_changes('oid8_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')"
+);
+like(
+ $result,
+ qr/data\[text\]:'(?:1234567890){2000}'/,
+ 'oid8 TOAST value past 2^32 is decoded in full');
+
# done with the node
$node_primary->stop;
--
2.55.0
v18-0005-Add-check-for-2-32-oid8-chunk-values-across-the-.patch (text/plain, 2.5K) download
From dee849e3d11995616a13cfcd88a5d319eb5d3915 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 14 Sep 2026 13:54:44 +0900
Subject: [PATCH v18 5/5] Add check for 2^32 oid8 chunk values across the main
regression test suite
As a matter of making this cheaper, this check is included in
pg_upgrade, where we also need to check that oid8 values are carried
across upgrades.
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 41 +++++++++++++++++---------
1 file changed, 27 insertions(+), 14 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 728fd6b0b120..f426074ce6bf 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -229,6 +229,21 @@ $oldnode->append_conf('postgresql.conf', 'log_statement = none');
# Set wal_level = replica to run the regression tests in the same
# wal_level as when 'make check' runs.
$oldnode->append_conf('postgresql.conf', 'wal_level = replica');
+
+# The OID counter is 8 bytes wide, check that it is carried across the
+# test. Older versions may not support 8-byte OIDs, so skip in this case.
+my $big_next_oid = '4295067296'; # 2^32 + 100000
+if (!defined($ENV{oldinstall}))
+{
+ command_ok(
+ [
+ 'pg_resetwal',
+ '--next-oid' => $big_next_oid,
+ $oldnode->data_dir
+ ],
+ 'set an 8-byte OID counter in the old instance');
+}
+
$oldnode->start;
my $result;
@@ -296,6 +311,18 @@ else
'regression tests in old instance');
}
+# Checks for 8-byte OIDs
+if (!defined($ENV{oldinstall}))
+{
+ # Table with 8-byte OID values, past 2^32.
+ is( $oldnode->safe_psql(
+ 'regression',
+ "SELECT max(pg_column_toast_chunk_id(f1)) > '$big_next_oid'::oid8 FROM toasttest_oid8"
+ ),
+ 't',
+ 'oid8 chunk_ids are past 2^32');
+}
+
# Initialize a new node for the upgrade.
my $newnode = PostgreSQL::Test::Cluster->new('new_node');
@@ -588,20 +615,6 @@ $oldnode->start;
$oldnode->safe_psql('postgres', 'DROP DATABASE regression_invalid');
$oldnode->stop;
-# The OID counter is 8 bytes wide, check that it is carried. Older versions
-# may not support 8-byte OIDs, so skip in this case.
-my $big_next_oid = '4295067296'; # 2^32 + 100000
-if (!defined($ENV{oldinstall}))
-{
- command_ok(
- [
- 'pg_resetwal',
- '--next-oid' => $big_next_oid,
- $oldnode->data_dir
- ],
- 'set an 8-byte OID counter in the old instance');
-}
-
# --check command works here, cleans up pg_upgrade_output.d.
command_ok(
[
--
2.55.0
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 05:12 Michael Paquier <michael@paquier.xyz>
parent: Marcos Pegoraro <marcos@f10.com.br>
0 siblings, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-14 05:12 UTC (permalink / raw)
To: Marcos Pegoraro <marcos@f10.com.br>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Sep 11, 2026 at 01:03:27PM -0300, Marcos Pegoraro wrote:
> Did you forget to mention the new REPACK command on create_table.SGML file ?
Yep, the statement is equally true with REPACK ("such as" means "like"
to me, meaning that the list could remain rather loose of details, but
I don't mind the addition).
I have amended the documentation. More importantly, I have also added
some tests in the v18 sent upthread, checking that the TOAST table
definition does not change across repack queries even after an ALTER
TABLE has the idea to change toast_value_type if a TOAST table already
exists.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 14:01 Greg Burd <greg@burd.me>
parent: Michael Paquier <michael@paquier.xyz>
3 siblings, 0 replies; 72+ messages in thread
From: Greg Burd @ 2026-09-14 14:01 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 14, 2026, at 1:00 AM, Michael Paquier wrote:
> On Fri, Sep 11, 2026 at 08:44:26AM -0700, Bharath Rupireddy wrote:
>> I reviewed v17 patches, overall they look good to me.
>>
>> v17-0001-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch: It
>> v17-0002-Switch-pg_column_toast_chunk_id-return-value-fro.patch: It
>
> I have discarded the change of reorderbuffer.c that was in v17-0001,
> and added it back to what's now v18-0001. Then applied these two
> after an extra round of self-review.
>
>> v17-0003-Add-support-for-oid8-TOAST-values.patch: It looks good to me
>> with one nit. Since chunk_id is always at attnum = 1 and all the
>> toast_valueid_scankey_init() callers pass it as 1, and this init
>> function is just to fetch the chunk_id, can we just hard-code it
>> inside and remove the attnum function parameter?
>
> If one wants to control which attribute number to look after, that
> could be extended more later on, yes. I have removed the argument
> from the function in v18-0001.
>
>> v17-0004-Add-battery-of-tests-related-oid8.patch: It looks good to me.
>
> Okay, thanks.
>
>> pg_dump/pg_restore for demoing OID4 to OID8 migration for
>> existing tables and TOAST table tests during pg_upgrade could be
>> follow-up patches. This makes me think, if someone does an OID8 to
>> OID4 migration and the chunk_ids are beyond the 4-billion limit, the
>> restore should fail rather than silently wrapping the chunk_ids and
>> causing TOAST index insert failures. My point is, we need to test this
>> case as well.
>
> Nah, I don't buy that this is worth the cycles. If one has the idea
> to create an oid8 TOAST table, inserts more than 4 billion values into
> it, dumps and restores into a table that uses a oid TOAST then the
> result is an infinite loop in the backend on restored INSERTs.
>
> Something that I still see as a hole is that we have the tables from
> the strings test in the main regression test suite and we could
> exploit that a bit more. Hmm. How about resetting the OID in the
> control file earlier in the pg_upgrade test and check with a SELECT
> post-upgrade that the counters generated by the main regression test
> suite are more than 2^32?
>
> Added also some tests for REPACK here in what's now 0002, based on the
> feedback of upthread, to check that the a REPACK's rewrite does not
> touch the TOAST table definition.
>
>> v17-0005-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
>
> And rebased is a v18 with the three remaining pieces, plus two more
> test patches: the tests of Rui with decoding and oid8 values higher
> than 2^32, and my test to have the main regression test suite of
> pg_upgrade run with a default oid8 in the control file higher than
> 2^32, checking that assigned chunk numbers are handled correctly.
> --
> Michael
>
> Attachments:
> * v18-0001-Add-support-for-oid8-TOAST-values.patch
> * v18-0002-Add-battery-of-tests-related-oid8.patch
> * v18-0003-Add-support-for-TOAST-pointers-as-oid8.patch
> * v18-0004-Decode-an-oid8-TOAST-value-with-an-ID-past-2-32.patch
> * v18-0005-Add-check-for-2-32-oid8-chunk-values-across-the-.patch
> * signature.asc
Hello Michael, all,
I tried v18 out, as expected patches/tests/etc. all good on my macOS/arm64
laptop.
I agree with Bharath that dropping the attnum argument from
toast_valueid_scankey_init() was right, and I see it is gone in 0001.
I like what you did in 0005 with the test, moving the pg_resetwal call ahead
of the main regression suite means the whole suite now runs with the counter
already past 2^32, rather than just one case checking a large value.
ok 1 - set an 8-byte OID counter in the old instance
ok 7 - oid8 chunk_ids are past 2^32
ok 25 - 8-byte OID counter is carried over by pg_upgrade
I'd say that closes the gap Bharath identified, where every oid8 test still
produced chunk_ids that fit in four bytes. Rui's decoding test in 0004 covers
the other half, and I checked separately that it fails when it should: with
ReorderBufferToastAppendChunk() put back to the truncating 4-byte read it
reports
not ok 23 - oid8 TOAST value past 2^32 is decoded in full
# table public.toasted_oid8: INSERT: id[integer]:1 data[text]:unchanged-toast-datum
and passes again once restored.
The REPACK additions in 0002 match what I measured by hand: with the reloption
flipped, VACUUM FULL, CLUSTER and REPACK all leave the TOAST table's chunk_id
type alone.
A quick check of my own on top of v18, counter preset to 5000000000: chunk_ids
5000000005 through 5000000054, all past 2^32, data intact, verify_heapam()
reports nothing on either the TOAST relation or the main table, and REPACK
leaves chunk_id as oid8 with the data still intact.
Nothing blocking from me on v18.
best.
-greg
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 19:26 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
3 siblings, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-14 19:26 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Sun, Sep 13, 2026 at 10:00 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> I have discarded the change of reorderbuffer.c that was in v17-0001,
> and added it back to what's now v18-0001. Then applied these two
> after an extra round of self-review.
Works for me.
> Added also some tests for REPACK here in what's now 0002, based on the
> feedback of upthread, to check that the a REPACK's rewrite does not
> touch the TOAST table definition.
>
> > v17-0005-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
>
> And rebased is a v18 with the three remaining pieces, plus two more
> test patches: the tests of Rui with decoding and oid8 values higher
> than 2^32, and my test to have the main regression test suite of
> pg_upgrade run with a default oid8 in the control file higher than
> 2^32, checking that assigned chunk numbers are handled correctly.
Nice! These are good additions IMV.
I reviewed v18 patches and I have no major comments.
v18-0001-Add-support-for-oid8-TOAST-values.patch: It looks good to me.
v18-0002-Add-battery-of-tests-related-oid8.patch: It looks good to me.
v18-0003-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
v18-0005-Add-check-for-2-32-oid8-chunk-values-across-the-.patch: It
looks good to me with a nit. It took me a while to figure out where
the f1 column is coming from, so worth adding a comment here (it is
the bytea column of the toasttest_oid8 table defined in
src/test/regress/sql/strings.sql).
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-14 23:43 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
3 siblings, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-14 23:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Sun, Sep 13, 2026 at 10:00 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Sep 11, 2026 at 08:44:26AM -0700, Bharath Rupireddy wrote:
>
> > pg_dump/pg_restore for demoing OID4 to OID8 migration for
> > existing tables and TOAST table tests during pg_upgrade could be
> > follow-up patches. This makes me think, if someone does an OID8 to
> > OID4 migration and the chunk_ids are beyond the 4-billion limit, the
> > restore should fail rather than silently wrapping the chunk_ids and
> > causing TOAST index insert failures. My point is, we need to test this
> > case as well.
>
> Nah, I don't buy that this is worth the cycles. If one has the idea
> to create an oid8 TOAST table, inserts more than 4 billion values into
> it, dumps and restores into a table that uses a oid TOAST then the
> result is an infinite loop in the backend on restored INSERTs.
>
> Something that I still see as a hole is that we have the tables from
> the strings test in the main regression test suite and we could
> exploit that a bit more. Hmm. How about resetting the OID in the
> control file earlier in the pg_upgrade test and check with a SELECT
> post-upgrade that the counters generated by the main regression test
> suite are more than 2^32?
I think I found an issue related to pg_dump/pg_restore where a dump
and restore after an ALTER TABLE RESET on the reloption would
downgrade the TOAST table from oid8 back to oid. More details in the
attached 0001 patch.
I also attached some more patches that I think would help with the
overall pg_dump and pg_restore along with the migration path for
existing oid TOAST tables.
I used the nocfbot prefix for these patches to not come in the way of
the main patches, and they can go after them. Please find them
attached.
0001 - fixes the pg_dump/pg_restore downgrade issue
0002 - adds a doc section on the migration
0003 - adds tests for the oid to oid8 migration and vice versa
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
Attachments: download all (3 files, .tar.gz)
nocfbot-v1-0001-pg_dump-Preserve-the-chunk_id-type-of-a-TOAST-rel.patch (application/octet-stream, 12.4K) download
From ca84a369ab3b76edaca98d1c9d07c3be1c011664 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 14 Sep 2026 18:43:33 +0000
Subject: [PATCH v1 1/3] pg_dump: Preserve the chunk_id type of a TOAST
relation.
toast_value_type is read only when a TOAST relation is created,
so a relation comes to use oid8 for its chunk_id by being created
with the option set.
Previously, pg_dump built the WITH clause of a relation from its
reloptions alone. A reset of toast_value_type removes the entry
from the reloptions of a relation but leaves its TOAST relation
alone, which keeps the oid8 chunk_id it was created with, and
pg_attribute is then the only place recording the type in use.
The dump of such a relation carried no type, and the restore
created its TOAST relation with oid instead.
Nothing reported an error or a warning, as the chunk identifiers
are assigned afresh by the restore, but the relation was limited
to 2^32 out-of-line values again.
Fix this by reporting the type of chunk_id in use when the
reloptions do not mention it, read from pg_attribute, so that a
plain dump carries it. A value present in the reloptions is a
request to change the type, and takes precedence.
Note that nothing is reported in binary upgrade mode. There the
type is carried by binary_upgrade_set_next_toast_chunk_id_typoid
and that path does not read the reloption at all, so pg_upgrade
preserved the type all along, and a WITH clause would only add an
entry the reloptions of the old cluster do not have.
The test sets the OID counter of the cluster past 2^32 before its
first start, so that losing oid8 is visible in the chunk
identifiers themselves and not only in the catalogs.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/aqd_gYs35v--sVlR%40paquier.xyz
---
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_dump.c | 47 ++++++-
src/bin/pg_dump/pg_dump.h | 2 +
src/bin/pg_dump/t/007_toast_value_type.pl | 150 ++++++++++++++++++++++
4 files changed, 199 insertions(+), 1 deletion(-)
create mode 100644 src/bin/pg_dump/t/007_toast_value_type.pl
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 79bd5036841..40521027576 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_toast_value_type.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 388c3b9c346..268a3440106 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -7187,6 +7187,7 @@ getTables(Archive *fout, int *numTables)
int i_reloptions;
int i_checkoption;
int i_toastreloptions;
+ int i_toastvaluetype;
int i_reloftype;
int i_foreignserver;
int i_amname;
@@ -7266,6 +7267,36 @@ getTables(Archive *fout, int *numTables)
"CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text "
"WHEN 'check_option=cascaded' = ANY (c.reloptions) THEN 'CASCADED'::text ELSE NULL END AS checkoption, ");
+ /*
+ * A reset of toast_value_type removes the entry from the reloptions of a
+ * relation but leaves its TOAST relation alone, which keeps the chunk_id
+ * type it was created with. The reloptions then say nothing about the
+ * type in use, and a WITH clause built from the reloptions alone would
+ * have the restore create the TOAST relation with the default type
+ * instead. Report the type of chunk_id for such a relation. Only oid8
+ * needs to be reported, as a relation using oid matches the default. A
+ * value present in the reloptions is a request to change the type, and
+ * takes precedence.
+ *
+ * Nothing is reported in binary upgrade mode. There the type is carried
+ * by binary_upgrade_set_next_toast_chunk_id_typoid and the reloption is
+ * not read at all, so a WITH clause would only add an entry the
+ * reloptions of the old cluster do not have.
+ */
+ if (fout->remoteVersion >= 190000 && !dopt->binary_upgrade)
+ appendPQExpBufferStr(query,
+ "CASE WHEN (SELECT a.atttypid FROM pg_attribute AS a "
+ " WHERE a.attrelid = c.reltoastrelid AND "
+ "attname = 'chunk_id'::text) = "
+ CppAsString2(OID8OID) " AND "
+ "NOT EXISTS (SELECT 1 FROM "
+ "unnest(coalesce(c.reloptions, '{}')) AS o "
+ "WHERE split_part(o, '=', 1) = 'toast_value_type') "
+ "THEN 'oid8' ELSE NULL END AS toast_value_type, ");
+ else
+ appendPQExpBufferStr(query,
+ "NULL AS toast_value_type, ");
+
appendPQExpBufferStr(query,
"am.amname, ");
@@ -7377,6 +7408,7 @@ getTables(Archive *fout, int *numTables)
i_reloptions = PQfnumber(res, "reloptions");
i_checkoption = PQfnumber(res, "checkoption");
i_toastreloptions = PQfnumber(res, "toast_reloptions");
+ i_toastvaluetype = PQfnumber(res, "toast_value_type");
i_reloftype = PQfnumber(res, "reloftype");
i_foreignserver = PQfnumber(res, "foreignserver");
i_amname = PQfnumber(res, "amname");
@@ -7458,6 +7490,10 @@ getTables(Archive *fout, int *numTables)
else
tblinfo[i].checkoption = pg_strdup(PQgetvalue(res, i, i_checkoption));
tblinfo[i].toast_reloptions = pg_strdup(PQgetvalue(res, i, i_toastreloptions));
+ if (PQgetisnull(res, i, i_toastvaluetype))
+ tblinfo[i].toast_value_type = NULL;
+ else
+ tblinfo[i].toast_value_type = pg_strdup(PQgetvalue(res, i, i_toastvaluetype));
tblinfo[i].reloftype = atooid(PQgetvalue(res, i, i_reloftype));
tblinfo[i].foreign_server = atooid(PQgetvalue(res, i, i_foreignserver));
if (PQgetisnull(res, i, i_amname))
@@ -17390,7 +17426,8 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
}
if (nonemptyReloptions(tbinfo->reloptions) ||
- nonemptyReloptions(tbinfo->toast_reloptions))
+ nonemptyReloptions(tbinfo->toast_reloptions) ||
+ tbinfo->toast_value_type != NULL)
{
bool addcomma = false;
@@ -17400,6 +17437,14 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
addcomma = true;
appendReloptionsArrayAH(q, tbinfo->reloptions, "", fout);
}
+ if (tbinfo->toast_value_type != NULL)
+ {
+ if (addcomma)
+ appendPQExpBufferStr(q, ", ");
+ addcomma = true;
+ appendPQExpBuffer(q, "toast_value_type=%s",
+ tbinfo->toast_value_type);
+ }
if (nonemptyReloptions(tbinfo->toast_reloptions))
{
if (addcomma)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 2bbb5d5773b..d3e6c1603bc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -315,6 +315,8 @@ typedef struct _tableInfo
char *reloptions; /* options specified by WITH (...) */
char *checkoption; /* WITH CHECK OPTION, if any */
char *toast_reloptions; /* WITH options for the TOAST table */
+ char *toast_value_type; /* type of chunk_id in the TOAST relation,
+ * when not implied by reloptions */
bool hasindex; /* does it have any indexes? */
bool hasrules; /* does it have any rules? */
bool hastriggers; /* does it have any triggers? */
diff --git a/src/bin/pg_dump/t/007_toast_value_type.pl b/src/bin/pg_dump/t/007_toast_value_type.pl
new file mode 100644
index 00000000000..ecec20f3201
--- /dev/null
+++ b/src/bin/pg_dump/t/007_toast_value_type.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Tests how a dump and restore treats the type a TOAST relation uses for its
+# chunk_id attribute. That type comes from the toast_value_type reloption of
+# the relation owning it, read only when the TOAST relation is created, so the
+# reloptions of a relation and the type its TOAST relation uses can differ.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# 2^32 + 100000.
+my $big_next_oid = '4295067296';
+
+# Type of the chunk_id attribute of the TOAST relation of a relation.
+sub chunk_id_type
+{
+ my ($node, $dbname, $relname) = @_;
+
+ return $node->safe_psql(
+ $dbname,
+ "SELECT a.atttypid::regtype FROM pg_class AS c, pg_attribute AS a
+ WHERE c.oid = '$relname'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'");
+}
+
+# Whether all the out-of-line values of a relation have a chunk_id that does
+# not fit in an oid. The threshold compared against is the OID counter set
+# below, which sits above the largest oid.
+sub chunk_ids_past_oid_max
+{
+ my ($node, $dbname, $relname) = @_;
+
+ return $node->safe_psql(
+ $dbname,
+ "SELECT bool_and(pg_column_toast_chunk_id(val) > '$big_next_oid'::oid8)
+ FROM $relname");
+}
+
+# Reloptions of a relation, as a string, or an empty string when it has none.
+sub reloptions
+{
+ my ($node, $dbname, $relname) = @_;
+
+ return $node->safe_psql(
+ $dbname,
+ "SELECT coalesce(array_to_string(reloptions, ','), '')
+ FROM pg_class WHERE oid = '$relname'::regclass");
+}
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+
+# Push the OID counter past 2^32 before the first start, so that the chunk
+# identifiers handed out in an oid8 TOAST relation do not fit in an oid. A
+# relation that loses oid8 along the way is then visible in the values
+# themselves and not just in the catalogs.
+command_ok([ 'pg_resetwal', '--next-oid' => $big_next_oid, $node->data_dir ],
+ 'set an OID counter past 2^32');
+
+$node->start;
+
+$node->safe_psql('postgres', 'CREATE DATABASE src');
+$node->safe_psql('postgres', 'CREATE DATABASE dst');
+
+# Test case 1: reloptions that say nothing about the type in use.
+#
+# A reset of toast_value_type removes the entry from the reloptions of a
+# relation but leaves its TOAST relation alone, which keeps the chunk_id type
+# it was created with.
+#
+# EXTERNAL storage keeps the values out of line without compressing them
+# first.
+$node->safe_psql(
+ 'src', qq{
+CREATE TABLE t_oid8_reset (id int, val text)
+ WITH (toast_value_type = 'oid8');
+ALTER TABLE t_oid8_reset ALTER COLUMN val SET STORAGE EXTERNAL;
+INSERT INTO t_oid8_reset
+ SELECT g, repeat('a', 5000) FROM generate_series(1, 3) g;
+});
+
+is(chunk_id_type($node, 'src', 't_oid8_reset'),
+ 'oid8', 't_oid8_reset: chunk_id is an oid8 when asked for at creation');
+is(chunk_ids_past_oid_max($node, 'src', 't_oid8_reset'),
+ 't', 't_oid8_reset: chunk identifiers do not fit in an oid');
+
+$node->safe_psql('src', 'ALTER TABLE t_oid8_reset RESET (toast_value_type)');
+
+is(reloptions($node, 'src', 't_oid8_reset'),
+ '', 't_oid8_reset: reloptions are empty after the reset');
+is(chunk_id_type($node, 'src', 't_oid8_reset'),
+ 'oid8', 't_oid8_reset: the reset does not change the TOAST relation');
+
+# Dump and restore.
+my $dumpfile = $node->basedir . '/toast_value_type.dump';
+my $textfile = $node->basedir . '/toast_value_type.sql';
+
+$node->command_ok(
+ [ 'pg_dump', '--format' => 'custom', '--file' => $dumpfile, 'src' ],
+ 'dump the source database');
+$node->command_ok([ 'pg_restore', '--file' => $textfile, $dumpfile ],
+ 'convert the dump to text');
+
+my $dump = slurp_file($textfile);
+
+like(
+ $dump,
+ qr/CREATE TABLE public\.t_oid8_reset \(\n[^)]*\)\nWITH \(toast_value_type=oid8\);/,
+ 't_oid8_reset: the dump asks for oid8');
+
+$node->command_ok(
+ [ 'pg_restore', '--dbname' => 'dst', $dumpfile ],
+ 'restore into the destination database');
+
+is(chunk_id_type($node, 'dst', 't_oid8_reset'),
+ 'oid8', 't_oid8_reset: the restored TOAST relation still uses oid8');
+is(chunk_ids_past_oid_max($node, 'dst', 't_oid8_reset'),
+ 't', 't_oid8_reset: the restored chunk identifiers do not fit in an oid');
+is( $node->safe_psql(
+ 'dst', 'SELECT count(*), min(length(val)) FROM t_oid8_reset'),
+ '3|5000',
+ 't_oid8_reset: the out-of-line values are restored intact');
+
+# A binary upgrade dump carries the type in a separate call, not a reloption.
+my $oid8_typoid = $node->safe_psql('src', "SELECT 'oid8'::regtype::oid");
+my $bufile = $node->basedir . '/toast_value_type_binary_upgrade.sql';
+
+$node->command_ok(
+ [
+ 'pg_dump', '--binary-upgrade', '--schema-only',
+ '--table' => 't_oid8_reset',
+ '--file' => $bufile,
+ 'src'
+ ],
+ 'dump the source database for a binary upgrade');
+
+my $budump = slurp_file($bufile);
+
+like(
+ $budump,
+ qr/binary_upgrade_set_next_toast_chunk_id_typoid\('$oid8_typoid'/,
+ 't_oid8_reset: a binary upgrade dump carries the type separately');
+unlike($budump, qr/toast_value_type/,
+ 't_oid8_reset: a binary upgrade dump reports no reloption for the type');
+
+done_testing();
--
2.47.3
nocfbot-v1-0002-doc-Describe-how-to-change-the-chunk_id-type-of-a.patch (application/octet-stream, 3.7K) download
From 3aa7120591117c08f5a07c8f1398eaf1ba65003a Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 14 Sep 2026 18:43:45 +0000
Subject: [PATCH v1 2/3] doc: Describe how to change the chunk_id type of a
TOAST table.
This commit documents the steps to change the chunk_id type of an
existing table, which takes a dump and restore rather than an
ALTER TABLE on its own.
A note covers the change back to oid, which reports no error even
for a table whose chunk identifiers did not fit in an oid, and
limits the table to 2^32 out-of-line values again.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/aqd_gYs35v--sVlR%40paquier.xyz
---
doc/src/sgml/storage.sgml | 65 +++++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 83de016eaa5..6b8fc2ed24a 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -595,6 +595,71 @@ tuple would otherwise be too big.
</sect2>
+<sect2 id="storage-toast-chunk-id-type">
+ <title>Changing the <structfield>chunk_id</structfield> Type of a TOAST Relation</title>
+
+<para>
+To change the <structfield>chunk_id</structfield> type of an existing table to
+<type>oid8</type>, set
+<link linkend="reloption-toast-value-type"><literal>toast_value_type</literal></link>
+and then dump and restore the table:
+<programlisting>
+ALTER TABLE mytable SET (toast_value_type = 'oid8');
+</programlisting>
+<programlisting>
+pg_dump -Fc -t mytable mydb > mytable.dump
+pg_restore -d mynewdb mytable.dump
+</programlisting>
+The <command>ALTER TABLE</command> only records the new value, which is
+consulted when a <acronym>TOAST</acronym> relation is created. The restore is
+what creates one and therefore applies it, so it has to be the command that
+creates the table. Restoring over a table that still exists leaves its
+<acronym>TOAST</acronym> relation alone, so drop the original first when
+changing a table in place.
+</para>
+
+<para>
+A table that has no <acronym>TOAST</acronym> relation yet takes no dump and
+restore, as the value recorded now is applied when a wide column added later
+causes one to be created.
+</para>
+
+<para>
+A table rewrite such as <command>CLUSTER</command>,
+<command>REPACK</command> or <command>VACUUM FULL</command> preserves the
+type already in use, and so does a <application>pg_upgrade</application>,
+which carries the existing <acronym>TOAST</acronym> relation over to the new
+cluster. The only way to recreate the <acronym>TOAST</acronym> relation of a
+table is to dump and restore it.
+</para>
+
+<para>
+The type in use by the <acronym>TOAST</acronym> relation of a table can be
+checked with:
+<programlisting>
+SELECT a.atttypid::regtype
+ FROM pg_class c, pg_attribute a
+ WHERE c.oid = 'mytable'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+</programlisting>
+</para>
+
+<note>
+<para>
+To change the <structfield>chunk_id</structfield> type of a table back to
+<type>oid</type>, set <literal>toast_value_type</literal> to
+<literal>oid</literal> and then dump and restore the table the same way. The
+chunk identifiers are assigned afresh during the restore, so the ones that did
+not fit in an <type>oid</type> are replaced by ones that do, and no error or
+warning is reported. Such a table is however limited again to 2^32
+out-of-line values, and as the OID space fills up, finding an OID that is
+still free when inserting into the <acronym>TOAST</acronym> relation can
+become expensive. The server log reports <quote>still searching for an unused
+OID in relation</quote> when that happens.
+</para>
+</note>
+</sect2>
+
</sect1>
<sect1 id="storage-fsm">
--
2.47.3
nocfbot-v1-0003-pg_dump-Test-a-requested-change-of-the-chunk_id-t.patch (application/octet-stream, 5.1K) download
From 7d58a2e36956dbcdbf4b277f67ddfa69fe1fd11c Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 14 Sep 2026 23:32:46 +0000
Subject: [PATCH v1 3/3] pg_dump: Test a requested change of the chunk_id type.
This commit extends the tests for the chunk_id type of a TOAST
relation to cover a change of the type in use by an existing
table, from oid to oid8 and the other way round.
The type only changes once the restore has recreated the TOAST
relation, so the checks run both before and after a dump and
restore.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/aqd_gYs35v--sVlR%40paquier.xyz
---
src/bin/pg_dump/t/007_toast_value_type.pl | 78 +++++++++++++++++++++++
1 file changed, 78 insertions(+)
diff --git a/src/bin/pg_dump/t/007_toast_value_type.pl b/src/bin/pg_dump/t/007_toast_value_type.pl
index ecec20f3201..9087403a6d4 100644
--- a/src/bin/pg_dump/t/007_toast_value_type.pl
+++ b/src/bin/pg_dump/t/007_toast_value_type.pl
@@ -95,6 +95,42 @@ is(reloptions($node, 'src', 't_oid8_reset'),
is(chunk_id_type($node, 'src', 't_oid8_reset'),
'oid8', 't_oid8_reset: the reset does not change the TOAST relation');
+# Test case 2: reloptions that specify a type.
+#
+# A table created with oid, whose out-of-line values get identifiers to match,
+# and a set of the option that records a request for oid8.
+$node->safe_psql(
+ 'src', qq{
+CREATE TABLE t_oid_to_oid8 (id int, val text);
+ALTER TABLE t_oid_to_oid8 ALTER COLUMN val SET STORAGE EXTERNAL;
+INSERT INTO t_oid_to_oid8
+ SELECT g, repeat('b', 5000) FROM generate_series(1, 3) g;
+ALTER TABLE t_oid_to_oid8 SET (toast_value_type = 'oid8');
+});
+
+# The set does not recreate the TOAST relation. The OID counter sits past
+# 2^32, yet an oid identifier still has to fit in one.
+is(chunk_id_type($node, 'src', 't_oid_to_oid8'),
+ 'oid', 't_oid_to_oid8: the set does not change the TOAST relation');
+is(chunk_ids_past_oid_max($node, 'src', 't_oid_to_oid8'),
+ 'f', 't_oid_to_oid8: chunk identifiers still fit in an oid');
+
+# The same the other way round, a table created with oid8 asking back for oid.
+$node->safe_psql(
+ 'src', qq{
+CREATE TABLE t_oid8_to_oid (id int, val text)
+ WITH (toast_value_type = 'oid8');
+ALTER TABLE t_oid8_to_oid ALTER COLUMN val SET STORAGE EXTERNAL;
+INSERT INTO t_oid8_to_oid
+ SELECT g, repeat('c', 5000) FROM generate_series(1, 3) g;
+ALTER TABLE t_oid8_to_oid SET (toast_value_type = 'oid');
+});
+
+is(chunk_id_type($node, 'src', 't_oid8_to_oid'),
+ 'oid8', 't_oid8_to_oid: the set does not change the TOAST relation');
+is(chunk_ids_past_oid_max($node, 'src', 't_oid8_to_oid'),
+ 't', 't_oid8_to_oid: chunk identifiers do not fit in an oid');
+
# Dump and restore.
my $dumpfile = $node->basedir . '/toast_value_type.dump';
my $textfile = $node->basedir . '/toast_value_type.sql';
@@ -111,6 +147,14 @@ like(
$dump,
qr/CREATE TABLE public\.t_oid8_reset \(\n[^)]*\)\nWITH \(toast_value_type=oid8\);/,
't_oid8_reset: the dump asks for oid8');
+like(
+ $dump,
+ qr/CREATE TABLE public\.t_oid_to_oid8 \(\n[^)]*\)\nWITH \(toast_value_type=oid8\);/,
+ 't_oid_to_oid8: the dump asks for oid8');
+like(
+ $dump,
+ qr/CREATE TABLE public\.t_oid8_to_oid \(\n[^)]*\)\nWITH \(toast_value_type=oid\);/,
+ 't_oid8_to_oid: the dump asks for oid');
$node->command_ok(
[ 'pg_restore', '--dbname' => 'dst', $dumpfile ],
@@ -125,6 +169,40 @@ is( $node->safe_psql(
'3|5000',
't_oid8_reset: the out-of-line values are restored intact');
+# The restore is what applies the request, and the values come over intact.
+is(chunk_id_type($node, 'dst', 't_oid_to_oid8'),
+ 'oid8', 't_oid_to_oid8: the restored TOAST relation uses oid8');
+is(chunk_ids_past_oid_max($node, 'dst', 't_oid_to_oid8'),
+ 't',
+ 't_oid_to_oid8: the restored chunk identifiers do not fit in an oid');
+is( $node->safe_psql(
+ 'dst', 'SELECT count(*), min(length(val)) FROM t_oid_to_oid8'),
+ '3|5000',
+ 't_oid_to_oid8: the out-of-line values are restored intact');
+
+# The same value stored again gets an identifier no oid could hold.
+$node->safe_psql('dst',
+ "INSERT INTO t_oid_to_oid8 SELECT 4, repeat('b', 5000)");
+
+is(chunk_ids_past_oid_max($node, 'dst', 't_oid_to_oid8'),
+ 't', 't_oid_to_oid8: a new identifier does not fit in an oid');
+
+# The other way round, the restored relation uses oid and so does a new value.
+is(chunk_id_type($node, 'dst', 't_oid8_to_oid'),
+ 'oid', 't_oid8_to_oid: the restored TOAST relation uses oid');
+is(chunk_ids_past_oid_max($node, 'dst', 't_oid8_to_oid'),
+ 'f', 't_oid8_to_oid: the restored chunk identifiers fit in an oid');
+is( $node->safe_psql(
+ 'dst', 'SELECT count(*), min(length(val)) FROM t_oid8_to_oid'),
+ '3|5000',
+ 't_oid8_to_oid: the out-of-line values are restored intact');
+
+$node->safe_psql('dst',
+ "INSERT INTO t_oid8_to_oid SELECT 4, repeat('c', 5000)");
+
+is(chunk_ids_past_oid_max($node, 'dst', 't_oid8_to_oid'),
+ 'f', 't_oid8_to_oid: a new identifier fits in an oid');
+
# A binary upgrade dump carries the type in a separate call, not a reloption.
my $oid8_typoid = $node->safe_psql('src', "SELECT 'oid8'::regtype::oid");
my $bufile = $node->basedir . '/toast_value_type_binary_upgrade.sql';
--
2.47.3
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-15 07:08 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 2 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-15 07:08 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 14, 2026 at 12:26:54PM -0700, Bharath Rupireddy wrote:
> v18-0001-Add-support-for-oid8-TOAST-values.patch: It looks good to me.
> v18-0002-Add-battery-of-tests-related-oid8.patch: It looks good to me.
> v18-0003-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
Thanks. I have spent the day processing all these free, and finally
applied the set.
One thing that neither you nor Greg pointed out is a disturbance I
have found in amcheck. The code relied on a vartag to decide the max
chunk size of a TOAST blob:
+ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8
+ ? TOAST_OID8_MAX_CHUNK_SIZE
+ : TOAST_OID_MAX_CHUNK_SIZE;
But that's wrong, because we want to check the sanity of the toast
pointer itself. If the tag is corrupted for a reason or another, we
could finish with an incorrect check. The correct way to handle this
case is to retrieve the max chunk based on the type of chunk_id as
defined in the cached toast_rel information, and also cross-check that
the vartag matches with the chunk_id type.
While on it, I have noticed a small mistake after commit 6e58d6356fcb
in check_toasted_attribute(): a return is missing after the first
report_toast_corruption(). Will adjust that..
> v18-0005-Add-check-for-2-32-oid8-chunk-values-across-the-.patch: It
> looks good to me with a nit. It took me a while to figure out where
> the f1 column is coming from, so worth adding a comment here (it is
> the bytea column of the toasttest_oid8 table defined in
> src/test/regress/sql/strings.sql).
v18-0004 and v18-0015 are not there yet.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-15 07:59 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 2 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-15 07:59 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 14, 2026 at 04:43:44PM -0700, Bharath Rupireddy wrote:
> I think I found an issue related to pg_dump/pg_restore where a dump
> and restore after an ALTER TABLE RESET on the reloption would
> downgrade the TOAST table from oid8 back to oid. More details in the
> attached 0001 patch.
I am not convinced by this argument. To me, applying the value set in
the reloption at the moment of the dump and not the chunk_id type
feels like a natural thing to do. We still have time to decide which
behavior makes more sense for this release, but choosing what the
reloption holds at the time of the dump is a property I'd prefer keep.
> I also attached some more patches that I think would help with the
> overall pg_dump and pg_restore along with the migration path for
> existing oid TOAST tables.
>
> I used the nocfbot prefix for these patches to not come in the way of
> the main patches, and they can go after them. Please find them
> attached.
>
> 0001 - fixes the pg_dump/pg_restore downgrade issue
> 0002 - adds a doc section on the migration
> 0003 - adds tests for the oid to oid8 migration and vice versa
0002 seems perhaps like something we should do, but it's independent
of 0001, no? Not sure that the tests of 0003 are really needed. That
feels like bloat compared to the rest already in place.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-15 15:26 Greg Burd <greg@burd.me>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 72+ messages in thread
From: Greg Burd @ 2026-09-15 15:26 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Sep 15, 2026, at 3:08 AM, Michael Paquier wrote:
> On Mon, Sep 14, 2026 at 12:26:54PM -0700, Bharath Rupireddy wrote:
>> v18-0001-Add-support-for-oid8-TOAST-values.patch: It looks good to me.
>> v18-0002-Add-battery-of-tests-related-oid8.patch: It looks good to me.
>> v18-0003-Add-support-for-TOAST-pointers-as-oid8.patch: It looks good to me.
>
Hello Michael,
> One thing that neither you nor Greg pointed out is a disturbance I
> have found in amcheck.
Fair hit, and thanks for catching it. I read that hunk across several
versions and never questioned the direction of the inference. Deriving the
expected chunk size from the vartag is backwards in a corruption checker,
since the vartag is part of what is under suspicion; the chunk_id type from
the cached toast_rel is the trustworthy side, with the vartag then checked
against it. I will be looking at amcheck differently from now on.
> To me, applying the value set in the reloption at the moment of the dump
> and not the chunk_id type feels like a natural thing to do. We still have
> time to decide which behavior makes more sense for this release, but
> choosing what the reloption holds at the time of the dump is a property
> I'd prefer keep.
I confirmed Bharath's case on f292187f375: create with oid8, load values with
the counter past 2^32, then
ALTER TABLE t8 RESET (toast_value_type);
leaves reloptions null while chunk_id stays oid8, pg_dump then emits no WITH
clause at all, and the restore comes back with an oid TOAST table. Data is
intact and the IDs are simply reassigned small, so on a table this size
nothing is lost.
The part that gives me pause is the interaction with the case you raised
upthread yourself: an oid8 table with more than four billion live values,
dumped and restored into an oid TOAST table, spins in
GetNewOidWithIndex(). RESET is the one step that gets a user there without
ever asking for it. The dump carries no trace of the property that was
dropped, so the first sign of trouble is a restore that does not finish, on
exactly the tables big enough to have needed oid8.
I do not think that argues against your semantics for the reloption in
general, which I agree read naturally. It argues that a reset on a relation
whose TOAST relation is already oid8 is the case worth treating specially.
One observation on Bharath's 0001 while I am here: I do not think it actually
conflicts with what you wrote. It applies the reloption when one is present,
and only reads pg_attribute when there is none, so "applying the value set in
the reloption at the moment of the dump" still holds under it. The two of you
differ over the reloption-absent case, where there is no value set to apply,
rather than over which of the two wins.
Attached is a much smaller thing, offered to make the discussion concrete
rather than as the answer. Report the loss at the point where it happens,
and leave pg_dump alone entirely.
=# ALTER TABLE t8 RESET (toast_value_type);
WARNING: TOAST relation of "t8" keeps "oid8" as the type of its "chunk_id"
DETAIL: The type in use is only read when the TOAST relation is created.
HINT: Set "toast_value_type" to "oid8" to have a dump and restore preserve it.
Only a reset naming toast_value_type reports; resetting other options on the
same relation stays quiet, as does a reset on a relation whose TOAST relation
uses oid, and the documented upgrade path of setting oid8 is untouched.
Worth noting that the tests you committed in c68cba09dd7 already reset the
option on toasttest_oid8, so the warning lands there and the expected output
records it, which is the whole change outside tablecmds.c.
If you would rather the dump carry the type, Bharath's 0001 is the better
patch and this one should be dropped. My preference is either of them over
the current behaviour. His 0002 on the migration docs seems worth having
either way, and independent as you say.
best.
-greg
Attachments:
0001-Warn-when-a-reset-of-toast_value_type-drops-oid8-fro.patch (application/octet-stream, 3.8K) download
From 63fb0ea13e86760ab0460bf7f3dd165baf32032d Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Tue, 15 Sep 2026 07:46:39 -0400
Subject: [PATCH] Warn when a reset of toast_value_type drops oid8 from a dump
toast_value_type is read only when a TOAST relation is created, so a
reset leaves the TOAST relation alone: a relation created with oid8
keeps it, and pg_attribute becomes the only place recording the type in
use. Nothing fails, and the chunk identifiers are assigned afresh on a
restore, but the dump carries no trace of the type that was dropped and
the relation comes back limited to 2^32 out-of-line values.
That matters because restoring an oid8 relation holding more than four
billion out-of-line values into an oid TOAST relation spins in
GetNewOidWithIndex(), so the first sign of trouble can be a restore
that does not finish, on exactly the relations big enough to have
wanted oid8.
Report the loss where it happens rather than leaving it to be found on
the restore. Only a reset naming toast_value_type is reported, so
resetting other options on the same relation stays quiet, as does a
reset on a relation whose TOAST relation uses oid.
This is deliberately the smallest change that removes the silence; it
does not touch what pg_dump emits.
Discussion: https://postgr.es/m/aqj6yd7ztQldv4Tu@paquier.xyz
---
src/backend/commands/tablecmds.c | 30 +++++++++++++++++++++++++++
src/test/regress/expected/strings.out | 3 +++
2 files changed, 33 insertions(+)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 2f073ddb84a..ba7e0931099 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17355,6 +17355,36 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation,
newOptions = transformRelOptions(datum, defList, NULL, validnsps, false,
operation == AT_ResetRelOptions);
+ /*
+ * A reset of toast_value_type leaves the TOAST relation alone, so a
+ * relation created with oid8 keeps it, and pg_attribute becomes the only
+ * place recording the type in use. Nothing fails, but a later dump and
+ * restore recreates the TOAST relation from the reloptions and so hands
+ * the relation back limited to 2^32 out-of-line values. Say so, since
+ * the loss is otherwise silent and only shows up on the restore.
+ */
+ if (operation == AT_ResetRelOptions &&
+ RelationGetToastChunkIdType(rel) == OID8OID)
+ {
+ ListCell *cell;
+
+ foreach(cell, defList)
+ {
+ DefElem *def = (DefElem *) lfirst(cell);
+
+ if (def->defnamespace != NULL ||
+ strcmp(def->defname, "toast_value_type") != 0)
+ continue;
+
+ ereport(WARNING,
+ (errmsg("TOAST relation of \"%s\" keeps \"oid8\" as the type of its \"chunk_id\"",
+ RelationGetRelationName(rel)),
+ errdetail("The type in use is only read when the TOAST relation is created."),
+ errhint("Set \"toast_value_type\" to \"oid8\" to have a dump and restore preserve it.")));
+ break;
+ }
+ }
+
/* Validate */
switch (rel->rd_rel->relkind)
{
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out
index fa29abfd829..902a3671e28 100644
--- a/src/test/regress/expected/strings.out
+++ b/src/test/regress/expected/strings.out
@@ -2304,6 +2304,9 @@ SELECT c1.relname, a.atttypid::regtype
ALTER TABLE toasttest_oid RESET (toast_value_type);
ALTER TABLE toasttest_oid8 RESET (toast_value_type);
+WARNING: TOAST relation of "toasttest_oid8" keeps "oid8" as the type of its "chunk_id"
+DETAIL: The type in use is only read when the TOAST relation is created.
+HINT: Set "toast_value_type" to "oid8" to have a dump and restore preserve it.
-- Reset column storage to its default
ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
--
2.50.1 (Apple Git-155)
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-15 16:20 Rui Zhao <zhaorui126@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 2 replies; 72+ messages in thread
From: Rui Zhao @ 2026-09-15 16:20 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi Michael,
The tests of my last mail (standby promotion, pg_upgrade with the counter
past 2^32, rewrites after ALTER TABLE SET (toast_value_type), logical
replication) give the same results on master at fb60892f40, and
check-world passes there with 0006 and 0007 applied.
> One thing that neither you nor Greg pointed out is a disturbance I
> have found in amcheck. The code relied on a vartag to decide the max
> chunk size of a TOAST blob:
1. Once 0004 and 0005 are in, amcheck is the one consumer of oid8
pointers left that no test runs with IDs past 2^32. 0006 adds a TAP
test for it.
check_heap.sql runs on the regression instance, whose counter is young,
and the pg_upgrade test, where the counter is moved, does not run
amcheck. 0006 moves the counter past 2^32 before the first start, fills
an oid8 table with uncompressed and compressed out-of-line values,
replaces and deletes some of them, and runs verify_heapam(check_toast
:= true) on the table, verify_heapam() on the TOAST table and
bt_index_parent_check(heapallindexed => true) on the TOAST index,
before and after VACUUM. If I cast the value ID that
check_toasted_attribute() passes to toast_valueid_scankey_init() to
Oid by hand, the test fails, with verify_heapam() reporting every
value:
toast value 4295067314 not found in toast table
2. The "compressed out-of-line values" of check_heap.sql are not out of
line. 0007 makes them larger.
The rows come from these statements of check_heap.sql:
ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
INSERT INTO test_toast_oid8 (a, b)
(SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
and repeat('xyzzy', 20000) compresses to 1161 bytes and stays in the
heap:
SELECT a, length(b), pg_column_size(b), pg_column_compression(b),
pg_column_toast_chunk_id(b)
FROM test_toast_oid8 WHERE a = 6;
a | length | pg_column_size | pg_column_compression | pg_column_toast_chunk_id
6 | 100000 | 1161 | pglz |
so rows 6 to 10 of test_toast_oid and test_toast_oid8 hold no TOAST
pointer, and verify_heapam(check_toast := true) has nothing to look up
for them. With repeat('xyzzy', 100000) the compressed value is 5735
bytes and goes out of line; 0007 also adds a SELECT showing that all
ten values of each table are out of line and five of them compressed.
> To me, applying the value set in the reloption at the moment of the
> dump and not the chunk_id type feels like a natural thing to do.
3. That works with either of two behaviors for ALTER TABLE, and the
current one is neither. I prefer a) and can write it.
Today SET and RESET (toast_value_type) change pg_class.reloptions and
nothing else, and no rewrite applies the new value: VACUUM FULL,
CLUSTER, REPACK and ALTER TABLE SET ACCESS METHOD rebuild the TOAST
table with the type it already has, since create_toast_table() takes
that type from the old TOAST table. A dump and restore is the only
thing that applies the reloption. Other heap reloptions are applied by
a rewrite; a table of 20000 rows with fillfactor:
default, 167 pages
SET (fillfactor = 20), 167 pages
VACUUM FULL, 870 pages
RESET (fillfactor) and VACUUM FULL, 167 pages
a) Keep SET and RESET as they are and let a rewrite apply the value,
as fillfactor does. A rewrite and a restore would then build the TOAST
table the same way, which is what the reloption already says it does.
Going to oid8 costs nothing there. Going to oid has to assign the
value IDs by probing for free ones, which slows down as the TOAST
table fills, so it is a usable operation for a table that fits with
room to spare and not for one that had a reason to be oid8. A restore
into an oid TOAST table makes the same assignment, so this is not
something a rewrite would introduce; neither route avoids it.
b) Reject RESET and have SET rewrite the table, as SET ACCESS METHOD
does. The reloption then always describes the type in use, whether it
is set or absent, which is why pg_dump can stay as it is. It puts a
rewrite behind an ALTER TABLE ... SET, which no other reloption does
today.
Either one settles what Bharath's 0001 reports: a dump follows the
reloption because that is what the reloption means. If the restore
side should be able to choose the type as well, the way to do that is
the one used for default_table_access_method, but I do not see the
need for it now.
Say which one you prefer and I will send a patch for it.
Regards,
Rui
Attachments: download all (2 files, .tar.gz)
0006-Check-oid8-TOAST-values-past-2-32-with-amcheck.patch (application/octet-stream, 4.2K) download
From 9565d27b351a8a6fce57eda0c4218028793981e7 Mon Sep 17 00:00:00 2001
From: Rui Zhao <zhaorui126@gmail.com>
Date: Tue, 15 Sep 2026 00:45:53 +0800
Subject: [PATCH 6/7] Check oid8 TOAST values past 2^32 with amcheck
The amcheck regression tests cover TOAST relations with oid8 chunk IDs,
but their OID counter is young, so the IDs they see fit in four bytes.
Move the counter past 2^32 in a TAP test and run verify_heapam() on a
table with such values and on its TOAST table, and bt_index_parent_check()
on the TOAST index, before and after VACUUM.
---
contrib/amcheck/meson.build | 1 +
contrib/amcheck/t/007_verify_oid8_toast.pl | 87 ++++++++++++++++++++++
2 files changed, 88 insertions(+)
create mode 100644 contrib/amcheck/t/007_verify_oid8_toast.pl
diff --git a/contrib/amcheck/meson.build b/contrib/amcheck/meson.build
index d5137ef691d..269529d0108 100644
--- a/contrib/amcheck/meson.build
+++ b/contrib/amcheck/meson.build
@@ -50,6 +50,7 @@ tests += {
't/004_verify_nbtree_unique.pl',
't/005_pitr.pl',
't/006_verify_gin.pl',
+ 't/007_verify_oid8_toast.pl',
],
},
}
diff --git a/contrib/amcheck/t/007_verify_oid8_toast.pl b/contrib/amcheck/t/007_verify_oid8_toast.pl
new file mode 100644
index 00000000000..b5f4c337fc5
--- /dev/null
+++ b/contrib/amcheck/t/007_verify_oid8_toast.pl
@@ -0,0 +1,87 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+#
+# Check a table whose TOAST values have oid8 IDs past 2^32, on the heap
+# side and on the TOAST side. The amcheck regression tests use a young
+# OID counter, so their oid8 values have IDs that fit in four bytes.
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+
+use Test::More;
+
+my $next_oid = '4295067296'; # 2^32 + 100000
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+command_ok(
+ [ 'pg_resetwal', '--next-oid' => $next_oid, $node->data_dir ],
+ 'set an 8-byte OID counter');
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION amcheck');
+
+# Out-of-line values, uncompressed and compressed, some of them replaced
+# or deleted so that the TOAST table holds dead chunks as well. The
+# compressed values are large enough to stay out of line once compressed.
+$node->safe_psql(
+ 'postgres', q(
+CREATE TABLE toast_oid8 (a int PRIMARY KEY, b text)
+ WITH (toast_value_type = 'oid8');
+ALTER TABLE toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;
+INSERT INTO toast_oid8
+ SELECT gs, repeat(md5(gs::text), 400) FROM generate_series(1, 5) gs;
+ALTER TABLE toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
+INSERT INTO toast_oid8
+ SELECT gs, repeat(md5(gs::text), 20000) FROM generate_series(6, 10) gs;
+UPDATE toast_oid8 SET b = repeat(md5('plugh'), 20000) WHERE a IN (3, 8);
+DELETE FROM toast_oid8 WHERE a IN (5, 10);
+));
+
+my $toast_table = $node->safe_psql('postgres',
+ "SELECT reltoastrelid::regclass FROM pg_class WHERE relname = 'toast_oid8'"
+);
+my $toast_index = $node->safe_psql('postgres',
+ "SELECT indexrelid::regclass FROM pg_index WHERE indrelid = '$toast_table'::regclass"
+);
+
+is( $node->safe_psql(
+ 'postgres',
+ "SELECT count(*), count(pg_column_toast_chunk_id(b)),
+ min(pg_column_toast_chunk_id(b)) > '$next_oid'::oid8,
+ count(*) FILTER (WHERE pg_column_compression(b) IS NOT NULL)
+ FROM toast_oid8"),
+ '8|8|t|5',
+ 'all values are out of line, with IDs past 2^32');
+
+sub check_all
+{
+ my ($stage) = @_;
+
+ is( $node->safe_psql(
+ 'postgres',
+ "SELECT count(*) FROM verify_heapam('toast_oid8', check_toast := true)"
+ ),
+ '0',
+ "verify_heapam reports nothing for the table and its TOAST values $stage"
+ );
+ is( $node->safe_psql(
+ 'postgres', "SELECT count(*) FROM verify_heapam('$toast_table')"),
+ '0',
+ "verify_heapam reports nothing for the TOAST table $stage");
+ is( $node->safe_psql(
+ 'postgres',
+ "SELECT bt_index_parent_check('$toast_index', heapallindexed => true)"
+ ),
+ '',
+ "bt_index_parent_check passes on the TOAST index $stage");
+}
+
+check_all('with dead chunks');
+$node->safe_psql('postgres', 'VACUUM toast_oid8');
+check_all('after VACUUM');
+
+$node->stop;
+
+done_testing();
--
2.43.7
0007-Store-the-compressed-values-of-the-amcheck-TOAST-tes.patch (application/octet-stream, 4.0K) download
From 9252c0423099a56538c04517db03b1e25118ece2 Mon Sep 17 00:00:00 2001
From: Rui Zhao <zhaorui126@gmail.com>
Date: Tue, 15 Sep 2026 00:51:00 +0800
Subject: [PATCH 7/7] Store the compressed values of the amcheck TOAST test out
of line
The rows meant to hold compressed out-of-line values in check_heap.sql
compress to 1161 bytes and stay in the heap, so the test does not see a
compressed TOAST pointer of either type. Make the values large enough,
and show that every value is out of line and half of them compressed.
---
contrib/amcheck/expected/check_heap.out | 20 +++++++++++++++++---
contrib/amcheck/sql/check_heap.sql | 14 +++++++++++---
2 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index 8686fba712d..a4c26dd4f87 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -209,13 +209,27 @@ INSERT INTO test_toast_oid (a, b)
(SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
INSERT INTO test_toast_oid8 (a, b)
(SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
--- Compressed out-of-line values.
+-- Compressed out-of-line values. These have to be large enough to stay
+-- out of line once compressed.
ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
INSERT INTO test_toast_oid (a, b)
- (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+ (SELECT gs, repeat('xyzzy', 100000) FROM generate_series(6,10) gs);
INSERT INTO test_toast_oid8 (a, b)
- (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+ (SELECT gs, repeat('xyzzy', 100000) FROM generate_series(6,10) gs);
+-- All values are out of line, half of them compressed.
+SELECT 'test_toast_oid' AS relname, count(pg_column_toast_chunk_id(b)) AS out_of_line,
+ count(pg_column_compression(b)) AS compressed FROM test_toast_oid
+UNION ALL
+SELECT 'test_toast_oid8', count(pg_column_toast_chunk_id(b)),
+ count(pg_column_compression(b)) FROM test_toast_oid8
+ ORDER BY 1;
+ relname | out_of_line | compressed
+-----------------+-------------+------------
+ test_toast_oid | 10 | 5
+ test_toast_oid8 | 10 | 5
+(2 rows)
+
SELECT c.relname, a.atttypid::regtype AS chunk_id_type
FROM pg_class AS c, pg_attribute AS a
WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index eb46fe71035..0d0dd9834b9 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -122,13 +122,21 @@ INSERT INTO test_toast_oid (a, b)
(SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
INSERT INTO test_toast_oid8 (a, b)
(SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);
--- Compressed out-of-line values.
+-- Compressed out-of-line values. These have to be large enough to stay
+-- out of line once compressed.
ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;
ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;
INSERT INTO test_toast_oid (a, b)
- (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+ (SELECT gs, repeat('xyzzy', 100000) FROM generate_series(6,10) gs);
INSERT INTO test_toast_oid8 (a, b)
- (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);
+ (SELECT gs, repeat('xyzzy', 100000) FROM generate_series(6,10) gs);
+-- All values are out of line, half of them compressed.
+SELECT 'test_toast_oid' AS relname, count(pg_column_toast_chunk_id(b)) AS out_of_line,
+ count(pg_column_compression(b)) AS compressed FROM test_toast_oid
+UNION ALL
+SELECT 'test_toast_oid8', count(pg_column_toast_chunk_id(b)),
+ count(pg_column_compression(b)) FROM test_toast_oid8
+ ORDER BY 1;
SELECT c.relname, a.atttypid::regtype AS chunk_id_type
FROM pg_class AS c, pg_attribute AS a
WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND
--
2.43.7
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-15 17:05 Greg Burd <greg@burd.me>
parent: Rui Zhao <zhaorui126@gmail.com>
1 sibling, 1 reply; 72+ messages in thread
From: Greg Burd @ 2026-09-15 17:05 UTC (permalink / raw)
To: Rui Zhao <zhaorui126@gmail.com>; Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Sep 15, 2026, at 12:20 PM, Rui Zhao wrote:
> Hi Michael,
>
> The tests of my last mail (standby promotion, pg_upgrade with the counter
> past 2^32, rewrites after ALTER TABLE SET (toast_value_type), logical
> replication) give the same results on master at fb60892f40, and
> check-world passes there with 0006 and 0007 applied.
>
>> One thing that neither you nor Greg pointed out is a disturbance I
>> have found in amcheck. The code relied on a vartag to decide the max
>> chunk size of a TOAST blob:
>
> 1. Once 0004 and 0005 are in, amcheck is the one consumer of oid8
> pointers left that no test runs with IDs past 2^32. 0006 adds a TAP
> test for it.
Hello Rui,
> If I cast the value ID that check_toasted_attribute() passes to
> toast_valueid_scankey_init() to Oid by hand, the test fails, with
> verify_heapam() reporting every value:
>
> toast value 4295067314 not found in toast table
+1, and thank you for showing it red as well as green. That is the right
follow-up to the thing Michael found in the same file, since the reason
nothing caught that was the same reason nothing would catch this: amcheck
never runs against a counter past 2^32.
> 2. The "compressed out-of-line values" of check_heap.sql are not out of
> line. 0007 makes them larger.
Good catch, and I can confirm the diagnosis. I can also tell you the fix is
not large enough, because the outcome depends on default_toast_compression
and your numbers are the pglz ones. Same two values, same
SET STORAGE EXTENDED, on one build:
method input stored placement
pglz repeat('xyzzy',20000) 1161 inline
pglz repeat('xyzzy',100000) 5735 out-of-line
lz4 repeat('xyzzy',20000) 415 inline
lz4 repeat('xyzzy',100000) 1983 inline
So 0007 fixes check_heap.sql on a pglz build and leaves it checking nothing
on an lz4 one, which is the default where lz4 is compiled in. Whatever the
size, a repeat() of a short string compresses far enough that the threshold
is a property of the compression method rather than of the test.
Something that resists compression avoids the question entirely. This goes
out of line under both methods:
(SELECT string_agg(md5(g::text), '') FROM generate_series(1, 2000) g)
pglz 64000 chars, stored 64000, out-of-line
lz4 64000 chars, stored 59738, out-of-line
Pinning default_toast_compression in the test would work too, though it costs
you the coverage of whichever method you did not pin.
> 3. That works with either of two behaviors for ALTER TABLE, and the
> current one is neither. I prefer a) and can write it.
> Today SET and RESET (toast_value_type) change pg_class.reloptions and
> nothing else, and no rewrite applies the new value [...] A dump and
> restore is the only thing that applies the reloption. Other heap
> reloptions are applied by a rewrite
This is a better diagnosis than the one I posted an hour ago. I treated
the surprise as a reporting problem and sent a warning for the reset case.
You have identified that the surprise is a consequence of this reloption
being the only one that no rewrite applies, which is the actual oddity.
The fillfactor page counts make that hard to argue with.
So please consider my patch withdrawn in favor of either of yours. Under b)
it is plainly unnecessary, since a rejected reset has nothing to warn about.
Under a) it is close to unnecessary too, because a user who resets then
rewrites has asked for the change and the two paths agree.
If it helps break the tie, I lean b), for the reason you give: the reloption
then always describes the type in use, whether set or absent. That is the
property that makes Bharath's 0001 unnecessary rather than merely
outvoted, and it is the one that removes the case I was worried about, where
a reset taken for an unrelated reason turns into a restore that will not
finish on a relation with more than four billion out-of-line values.
The cost you name is real, a rewrite behind ALTER TABLE ... SET where no
other reloption does that. I would take it, because SET ACCESS METHOD is
already precedent for exactly that shape, and because an expensive operation
the user asked for beats a cheap one whose consequence shows up later
somewhere else. a) is still a clear improvement on today and I would not
object to it.
Either way this is Michael's call to make, I'm anxious to hear his thoughts
on this.
best.
-greg
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-15 22:22 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-15 22:22 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Tue, Sep 15, 2026 at 12:59 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Mon, Sep 14, 2026 at 04:43:44PM -0700, Bharath Rupireddy wrote:
> > I think I found an issue related to pg_dump/pg_restore where a dump
> > and restore after an ALTER TABLE RESET on the reloption would
> > downgrade the TOAST table from oid8 back to oid. More details in the
> > attached 0001 patch.
>
> I am not convinced by this argument. To me, applying the value set in
> the reloption at the moment of the dump and not the chunk_id type
> feels like a natural thing to do. We still have time to decide which
> behavior makes more sense for this release, but choosing what the
> reloption holds at the time of the dump is a property I'd prefer keep.
>
> > I also attached some more patches that I think would help with the
> > overall pg_dump and pg_restore along with the migration path for
> > existing oid TOAST tables.
> >
> > I used the nocfbot prefix for these patches to not come in the way of
> > the main patches, and they can go after them. Please find them
> > attached.
> >
> > 0001 - fixes the pg_dump/pg_restore downgrade issue
> > 0002 - adds a doc section on the migration
> > 0003 - adds tests for the oid to oid8 migration and vice versa
>
> 0002 seems perhaps like something we should do, but it's independent
> of 0001, no? Not sure that the tests of 0003 are really needed. That
> feels like bloat compared to the rest already in place.
Firstly, thanks for fixing the thinkos in 0d34736e4c1. I will be more
careful and try to catch these next time.
Secondly, as far as these 3 patches are concerned, feel free to ignore
0003 for sure, as 0002 now pretty much covers what I wanted to say for
pg_dump and pg_restore. 0002 is something I agree we need to have. For
0001, I'm not sure the proposed fix is the best option we have here,
and I haven't read the other patches or thoughts on this problem yet.
But I do think the behavior needs to be addressed somehow, because
resetting the reloption is something one can easily do and later run
into issues or miss them entirely. Maybe just documenting it in 0002's
new section is simple enough, asking users to set the reloption back
correctly before running pg_dump.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-16 00:29 Michael Paquier <michael@paquier.xyz>
parent: Rui Zhao <zhaorui126@gmail.com>
1 sibling, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-16 00:29 UTC (permalink / raw)
To: Rui Zhao <zhaorui126@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Sep 16, 2026 at 12:20:28AM +0800, Rui Zhao wrote:
>> One thing that neither you nor Greg pointed out is a disturbance I
>> have found in amcheck. The code relied on a vartag to decide the max
>> chunk size of a TOAST blob:
(Cut some of the contents, some are clearly AI-generated, which is
slightly annoying to argue with.)
> 1. Once 0004 and 0005 are in, amcheck is the one consumer of oid8
> pointers left that no test runs with IDs past 2^32. 0006 adds a TAP
> test for it.
>
> If I cast the value ID that
> check_toasted_attribute() passes to toast_valueid_scankey_init() to
> Oid by hand, the test fails, with verify_heapam() reporting every
> value:
I'm wondering if we should think a bit more wildly here. For example,
we could have a new PG_TEST_EXTRA value that enforces an initial OID
beyong the 2^32 limit. This is a bit more general than the statement
you are making here. Just to say that I'm not much a fan of the
cycles spend on a new TAP test for this purpose, where installcheck
could do the same using the SQL queries. So I don't think that your
0006 buys much by just adding more cycles and spawning a new node for
what's limited and duplicated coverage compared to the SQLs of
amcheck/sql/. An advantage of dumping the OID up in the pg_upgrade
test is that it's basically free.
> 2. The "compressed out-of-line values" of check_heap.sql are not out of
> line. 0007 makes them larger.
>
> so rows 6 to 10 of test_toast_oid and test_toast_oid8 hold no TOAST
> pointer, and verify_heapam(check_toast := true) has nothing to look up
> for them. With repeat('xyzzy', 100000) the compressed value is 5735
> bytes and goes out of line; 0007 also adds a SELECT showing that all
> ten values of each table are out of line and five of them compressed.
Hmpf, yeah. These tuples are not chunked into TOAST as they compress
too well. There are a few ways to make that predictible and cheaper:
- Force default_toast_compression to pglz, like in portals.sql, and to
force a check based on pg_column_compression().
- Use the same trick as test_decoding's toast.sql, for a much faster
query and the same coverage and even less tuples.
There is a similar mistake in test_decoding/sql/toast.sql, for the
same assumptions. Will adjust both the same way, both tests want the
same data patterns. Thanks for the report.
>> To me, applying the value set in the reloption at the moment of the
>> dump and not the chunk_id type feels like a natural thing to do.
(Note: this is a point of design I've had in mind for close to 2
years, so it's not like I gave it no thoughts.)
> a) Keep SET and RESET as they are and let a rewrite apply the value,
> as fillfactor does. A rewrite and a restore would then build the TOAST
> table the same way, which is what the reloption already says it does.
And you're missing a very fancy point here. A user could be trapped
under a oid8->oid switch. Create an oid8 table, downgrade the
reloption to oid, do a VACUUM FULL and enjoy the infinite loop.
That's no different from somebody currently seeing the infinite 4
billion limitation with an oid table, but it surely hurts when
happening across a rewrite. toast_tuple_target and fillfactor impact
the heap pages and are per-write targets for future tuples written,
something like a VACUUM feeds on. Shaping a TOAST table based on a
reloption is an entirely new design space. It just does not have to
fit in these rules (this does not mean that I don't mind changing some
of the rules of HEAD, depending on how things go in this release
cycle).
> b) Reject RESET and have SET rewrite the table, as SET ACCESS METHOD
> does. The reloption then always describes the type in use, whether it
> is set or absent, which is why pg_dump can stay as it is. It puts a
> rewrite behind an ALTER TABLE ... SET, which no other reloption does
> today.
First. ALTER TABLE SET (reloption=foo) should never trigger a
rewrite. It's cheap currently as it just touches catalogs, and it
should stay as such. SET ACCESS METHOD is a different sub-command in
the ALTER TABLE set of options available. Whether it triggers a
rewrite is up to it because there is a user intention behind the cost.
A similar argument exists for SET LOGGED.
Anyway, this line of thought discards the fact that toast_value_type
defines the definition of an underlying table. Having a rewrite
enforced by a plain ALTER TABLE SET can also be surprising, especially
if you are dealing with a 10TB table that needs to be WAL-logged.
FWIW, we've had exactly a very similar kind of discussion in the v14
development when lz4 compression has been introduced for VACUUM
(outremembering a bunch of folks on this thread, I guess). The
original feature enforced a rewrite of the TOAST values based on the
default_toast_compression if the default was switched to lz4 while
values could be written as pglz, and the conclusion was that it was
not VACUUM's job to enforce new compression policies.
> Either one settles what Bharath's 0001 reports: a dump follows the
> reloption because that is what the reloption means. If the restore
> side should be able to choose the type as well, the way to do that
> is the one used for default_table_access_method, but I do not see
> the need for it now.
Among the options presented, preventing a RESET of the new reloption
if the reloption has been explicitely set may be OK, but it can be
disruptive with dump and upgrade scenarios in general, which is a
reason why it's originally pretty loose. Bharath's suggestion of
adding a WITH clause in the dumps based on the type of chunk_id when
taking the dump sounds like the most tempting semantic to me, because
it sticks the TOAST definition within the CREATE TABLE of the parent.
The devil is in the details, and there's still plenty of time to tweak
the semantics.
A GUC was shut down two years ago by Andres and Tom, see this thread
for a summary (feeding that to an agent would be faster, I guess):
https://www.postgresql.org/message-id/aFOnKHG7Wn-Srnpv@paquier.xyz
So I'm not going back to anything like that.
The rewrite of the TOAST table is something that I think would fit
much better on an option-basis in one of the existing commands, not
enforced upon users, *if and only if* explicitely requested. REPACK
would be a nice fit, in spirit, to be able to perform the job,
combined with an update of the reloption. VACUUM and CLUSTER are
there for different purposes. There would be value for such a thing
for a oid8->oid as much as an oid->oid8. At the end, it's also up to
users to be very careful. Not sure there is much a point to outsmart
them with the backend.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-16 00:34 Michael Paquier <michael@paquier.xyz>
parent: Greg Burd <greg@burd.me>
0 siblings, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-16 00:34 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Rui Zhao <zhaorui126@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Sep 15, 2026 at 01:05:09PM -0400, Greg Burd wrote:
> Pinning default_toast_compression in the test would work too, though it costs
> you the coverage of whichever method you did not pin.
Using default_toast_compression is perhaps nicer (same as
portals.sql). It offers more predictibility in the output and we
don't really care about the compression method in this case. The md5
computations can be more expensive, which I find annoying, so I'm
eager to just reuse the smart pattern of toast.sql instead. It's much
cheaper, and enforces the out-of-line behavior we want here.
Note: test_decoding has the same issue. Oops. Will fix in a bit
after more checks.
> Either way this is Michael's call to make, I'm anxious to hear his thoughts
> on this.
Replied to that upthread, as of:
https://www.postgresql.org/message-id/aqni9bvWV0UIUEZy@paquier.xyz
I don't think that this new reloption should equal in expectations
with fillfactor and/or tuple_toast_target.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-16 01:59 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
3 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-16 01:59 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 14, 2026 at 02:00:49PM +0900, Michael Paquier wrote:
> And rebased is a v18 with the three remaining pieces, plus two more
> test patches: the tests of Rui with decoding and oid8 values higher
> than 2^32, and my test to have the main regression test suite of
> pg_upgrade run with a default oid8 in the control file higher than
> 2^32, checking that assigned chunk numbers are handled correctly.
While thinking again about v18-0004 and v18-0005 that were the latest
items on my list, I am having second thoughts about v18-0004 that has
the idea to plug in one more pg_resetwal call to satisfy *ONE* case,
while the SQLs of test_decoding are able to do much more than that.
As a whole, it feels like a waste of cycles once we become able to do
an installcheck on a cluster that has an initial OID set to more than
UINT32_MAX. The pg_resetwal calls in pg_wal and the TLI recovery test
still make sense, as they cover specifics for the WAL records and the
control file. Just that the SQL tests are harder to justify presented
this way. So how about this wilder idea of a new PG_TEST_EXTRA
enforcing the initial OID value after an initdb?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-16 17:48 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-16 17:48 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Tue, Sep 15, 2026 at 6:59 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Mon, Sep 14, 2026 at 02:00:49PM +0900, Michael Paquier wrote:
> > And rebased is a v18 with the three remaining pieces, plus two more
> > test patches: the tests of Rui with decoding and oid8 values higher
> > than 2^32, and my test to have the main regression test suite of
> > pg_upgrade run with a default oid8 in the control file higher than
> > 2^32, checking that assigned chunk numbers are handled correctly.
>
> While thinking again about v18-0004 and v18-0005 that were the latest
> items on my list, I am having second thoughts about v18-0004 that has
> the idea to plug in one more pg_resetwal call to satisfy *ONE* case,
> while the SQLs of test_decoding are able to do much more than that.
>
> As a whole, it feels like a waste of cycles once we become able to do
> an installcheck on a cluster that has an initial OID set to more than
> UINT32_MAX. The pg_resetwal calls in pg_wal and the TLI recovery test
> still make sense, as they cover specifics for the WAL records and the
> control file. Just that the SQL tests are harder to justify presented
> this way. So how about this wilder idea of a new PG_TEST_EXTRA
> enforcing the initial OID value after an initdb?
While I agree with the general idea that having a way to enforce the
initial OID after an initdb is a good one, just curious, why is this
specific to the pg_resetwal being added to the recovery test in
006_logical_decoding.pl? Or is the intention to not exercise this test
always? I'm still not clear on why we need to solve this problem here.
Also, the fact that this test is kept at the end of the test file
means it doesn't impact any other test case in the file, though
keeping it at the end in the future could be hard if that's an issue
at all.
That said, I measured the cost that this additional pg_resetwal brings
to the test timings. On a release build on my dev machine (EC2
r7i.8xlarge), over 10 runs of just 006_logical_decoding.pl, the
testing time goes from 1.101s to 1.162s, of which the pg_resetwal
itself is about 29ms. This isn't huge IMO for bigger machines, however
this could be costlier for most of the CF bot, CI and BF animals out
there though.
In short, I don't think we need to worry much about this as part of
this thread. Maybe separately, yes, if the goal is to remove
pg_resetwal across all TAP tests with what you proposed above, that
seems fine to me.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-16 23:50 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-16 23:50 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Sep 16, 2026 at 10:48:35AM -0700, Bharath Rupireddy wrote:
> While I agree with the general idea that having a way to enforce the
> initial OID after an initdb is a good one, just curious, why is this
> specific to the pg_resetwal being added to the recovery test in
> 006_logical_decoding.pl? Or is the intention to not exercise this test
> always? I'm still not clear on why we need to solve this problem here.
The duplication at SQL level in the TAP tests that all these
pg_resetwal commands create is what drives me toward a more general
solution as being suited. We want to check for UNIT32_MAX overflows
in a general way than just something inside one test. Using a
pg_resetwal command in the pg_upgrade test feels like a good
investment: we run the main regression test suite all the time there.
Instead of a PG_TEST_EXTRA variable another set of ideas one I can
think of out-of-the-box is something that would be based on a compile
flag. For example a development GUC?
Or maybe just a few feature in itself, like an initdb option that
forces an initial OID after bootstrap in the control file? Then link
it to a new PG_TEST_EXTRA mode? Having something that could be
directly used by the buildfarm on an opt-in basis would be kind of
nice.
So, I'd rather have a more general mechanism in place rather than
finishing with 250 pg_resetwal commands sprinkled across the tree that
duplicate SQL tests we have in the script in 10 years.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-17 19:03 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-17 19:03 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Wed, Sep 16, 2026 at 4:50 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Sep 16, 2026 at 10:48:35AM -0700, Bharath Rupireddy wrote:
> > While I agree with the general idea that having a way to enforce the
> > initial OID after an initdb is a good one, just curious, why is this
> > specific to the pg_resetwal being added to the recovery test in
> > 006_logical_decoding.pl? Or is the intention to not exercise this test
> > always? I'm still not clear on why we need to solve this problem here.
>
> The duplication at SQL level in the TAP tests that all these
> pg_resetwal commands create is what drives me toward a more general
> solution as being suited. We want to check for UNIT32_MAX overflows
> in a general way than just something inside one test. Using a
> pg_resetwal command in the pg_upgrade test feels like a good
> investment: we run the main regression test suite all the time there.
>
> Instead of a PG_TEST_EXTRA variable another set of ideas one I can
> think of out-of-the-box is something that would be based on a compile
> flag. For example a development GUC?
>
> Or maybe just a few feature in itself, like an initdb option that
> forces an initial OID after bootstrap in the control file? Then link
> it to a new PG_TEST_EXTRA mode? Having something that could be
> directly used by the buildfarm on an opt-in basis would be kind of
> nice.
>
> So, I'd rather have a more general mechanism in place rather than
> finishing with 250 pg_resetwal commands sprinkled across the tree that
> duplicate SQL tests we have in the script in 10 years.
IIUC, the SQL duplication referred to is that the 8-byte TOAST
chunk_id tests proposed here repeat the same create table and DMLs
that toast.sql already has. I would like to keep things minimal to the
core where possible and have the 8-byte TOAST tests run every time
across various BF animals (not behind a PG_TEST_EXTRA flag or such).
A developer GUC behind a compile-time macro (similar to clobber cache
for example) would be a nice option. However, this doesn't meet my
requirement of running these tests every time on all BF animals. And I
don't think an initdb option is a good idea here, because this is not
something one would use in production, just for testing, so -1 on
that. Another approach, which I don't recommend, is to have a SQL
function that modifies the OID counter directly. I know this is much
worse because such a function, even behind a superuser check, is
reachable on production systems. No go here, but just saying.
That said, if I relax my requirement of running these tests every time
on every BF animal a bit, the developer GUC behind a compile-time
macro seems like a good idea.
Another approach is a new test module in src/test/modules (similar to
the xid_wraparound module) with a simple SQL function that advances
the OID counter in shared memory, similar to core's SetNextObjectId().
Being a test module, it doesn't impact production systems, and all the
8-byte TOAST chunk_id related tests added for this feature can be
moved here and run across all BF animals.
Others may have a different opinion. Meanwhile, IMHO, the remaining
v18 tests can go in, with this discussed separately.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-19 00:03 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-19 00:03 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Sep 17, 2026 at 12:03:00PM -0700, Bharath Rupireddy wrote:
> Another approach is a new test module in src/test/modules (similar to
> the xid_wraparound module) with a simple SQL function that advances
> the OID counter in shared memory, similar to core's SetNextObjectId().
> Being a test module, it doesn't impact production systems, and all the
> 8-byte TOAST chunk_id related tests added for this feature can be
> moved here and run across all BF animals.
That could be one thing. Yes. Or perhaps not.
> Others may have a different opinion. Meanwhile, IMHO, the remaining
> v18 tests can go in, with this discussed separately.
OK, you're right that it also sucks to do nothing for the logical
decoding. So I think that I'll just do something for the
test_decoding TAP test. I am less convinced about the dump/restore
tests you are proposing; these are utterly bloated.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-19 00:26 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-19 00:26 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Fri, Sep 18, 2026 at 5:03 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Sep 17, 2026 at 12:03:00PM -0700, Bharath Rupireddy wrote:
> > Another approach is a new test module in src/test/modules (similar to
> > the xid_wraparound module) with a simple SQL function that advances
> > the OID counter in shared memory, similar to core's SetNextObjectId().
> > Being a test module, it doesn't impact production systems, and all the
> > 8-byte TOAST chunk_id related tests added for this feature can be
> > moved here and run across all BF animals.
>
> That could be one thing. Yes. Or perhaps not.
>
> > Others may have a different opinion. Meanwhile, IMHO, the remaining
> > v18 tests can go in, with this discussed separately.
>
> OK, you're right that it also sucks to do nothing for the logical
> decoding. So I think that I'll just do something for the
> test_decoding TAP test.
Works for me. Thanks.
> I am less convinced about the dump/restore
> tests you are proposing; these are utterly bloated.
Out of the 3 patches I posted here:
https://postgr.es/m/CALj2ACUmBhBGb+4d8SRq9JT_ZsObCAvY6-10CKgSWX98zdv2DQ@mail.gmail.com,
I'm fine to drop 0003 (I don't have a strong opinion on that) as
mentioned here:
https://postgr.es/m/CALj2ACWgpn_aWzzagrerFqD0p1kQv_iioWyJB5R8Cu8j3_yqdA%40mail.gmail.com
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-21 11:11 John Naylor <johncnaylorls@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 72+ messages in thread
From: John Naylor @ 2026-09-21 11:11 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Sep 15, 2026 at 2:08 PM Michael Paquier <michael@paquier.xyz> wrote:
> Thanks. I have spent the day processing all these free, and finally
> applied the set.
Nice work!
It looks like doc/src/sgml/limits.sgml needs to be updated as well.
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-21 22:39 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: John Naylor <johncnaylorls@gmail.com>
0 siblings, 2 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-21 22:39 UTC (permalink / raw)
To: John Naylor <johncnaylorls@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Mon, Sep 21, 2026 at 4:11 AM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Tue, Sep 15, 2026 at 2:08 PM Michael Paquier <michael@paquier.xyz> wrote:
> > Thanks. I have spent the day processing all these free, and finally
> > applied the set.
>
> Nice work!
>
> It looks like doc/src/sgml/limits.sgml needs to be updated as well.
Nice catch. Thanks for pointing it out. Please find the attached patch for that.
I also attached the 0001 and 0002 from
https://www.postgresql.org/message-id/CALj2ACUmBhBGb%2B4d8SRq9JT\_ZsObCAvY6-10CKgSWX98zdv2DQ%40mail.gmail.com
here as 0002 and 0003.
In the attached 0003, I fixed an issue with the remote version check
and simplified the tests by reusing an existing table whose reloption
is already reset, and verifying that the dump reports the original
TOAST type. This is good enough to cover the new code without starting
a new instance just to test this. I also used a join instead of a
per-relation lookup in the query to get the TOAST type, which avoids
an extra catalog lookup for every table.
Please have a look.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
Attachments: download all (3 files, .tar.gz)
v1-0001-doc-Note-the-oid8-out-of-line-value-limit-in-the-.patch (application/octet-stream, 2.3K) download
From bfce901044c607a652beb2089b7fb9e49f4c7046 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 21 Sep 2026 10:39:33 -0700
Subject: [PATCH v1 1/3] doc: Note the oid8 out-of-line value limit in the
limits appendix.
The out-of-line values paragraph described only the 32-bit OID
case. Note that a TOAST table using an OID8 can store up to 2^64
values, with new identifiers assigned without searching for a
free one, and that it does not hit the OID-space slowdown. Also
mention the 22-byte TOAST pointer next to the existing 18-byte
one.
Reported-by: John Naylor <johncnaylorls@gmail.com>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/CANWCAZa%2BpWZL1Zjep1pz1F0sVWwNOQf%2BQfGErOaiUvC10Yr2XA%40mail.gmail.com
---
doc/src/sgml/limits.sgml | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/limits.sgml b/doc/src/sgml/limits.sgml
index f26f4466719..90c3fb08c5f 100644
--- a/doc/src/sgml/limits.sgml
+++ b/doc/src/sgml/limits.sgml
@@ -123,10 +123,11 @@
Variable-length fields of
types such as <type>text</type>, <type>varchar</type>, and <type>char</type>
can have their values stored out of line in the table's TOAST table when the
- values are large enough to require it. Only an 18-byte pointer must remain
- inside the tuple in the table's heap. For shorter length variable-length
- fields, either a 4-byte or 1-byte field header is used and the value is
- stored inside the heap tuple.
+ values are large enough to require it. Only an 18-byte or 22-byte pointer
+ must remain inside the tuple in the table's heap, for a TOAST table using an
+ OID or an OID8 respectively. For shorter length variable-length fields,
+ either a 4-byte or 1-byte field header is used and the value is stored
+ inside the heap tuple.
</para>
<para>
@@ -145,5 +146,8 @@
can become expensive, in turn slowing down INSERT/UPDATE statements.
Typically, this is only an issue for tables containing many terabytes
of data; partitioning is a possible workaround.
+ A TOAST table using an OID8 does not have this issue, as it can store up
+ to 2^64 out-of-line values and new identifiers are assigned without
+ searching for a free one; see <xref linkend="reloption-toast-value-type"/>.
</para>
</appendix>
--
2.47.3
v1-0002-doc-Describe-how-to-change-the-chunk_id-type-of-a.patch (application/octet-stream, 3.7K) download
From 64d50145f69faa53199e2f6d967e14c79a5cfba7 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 14 Sep 2026 18:43:45 +0000
Subject: [PATCH v1 2/3] doc: Describe how to change the chunk_id type of a
TOAST table.
Previously, there was no documented way to change the chunk_id
type of an existing table. This commit documents the steps, which
take a dump and restore rather than an ALTER TABLE on its own.
A note covers the change back to OID, which reports no error even
for a table whose chunk identifiers did not fit in an OID, and
limits the table to 2^32 out-of-line values again.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/aqd_gYs35v--sVlR%40paquier.xyz
---
doc/src/sgml/storage.sgml | 65 +++++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 83de016eaa5..437d467d3fb 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -595,6 +595,71 @@ tuple would otherwise be too big.
</sect2>
+<sect2 id="storage-toast-chunk-id-type">
+ <title>Changing the <structfield>chunk_id</structfield> Type of a TOAST Table</title>
+
+<para>
+To change the <structfield>chunk_id</structfield> type of an existing table
+to OID8, set
+<link linkend="reloption-toast-value-type"><literal>toast_value_type</literal></link>
+and then dump and restore the table:
+<programlisting>
+ALTER TABLE mytable SET (toast_value_type = 'oid8');
+</programlisting>
+<programlisting>
+pg_dump -Fc -t mytable mydb > mytable.dump
+pg_restore -d mynewdb mytable.dump
+</programlisting>
+The <command>ALTER TABLE</command> only records the new value, which is
+consulted when a <acronym>TOAST</acronym> table is created. The restore is
+what creates one and therefore applies it, so it has to be the command that
+creates the table. Restoring over a table that still exists leaves its
+<acronym>TOAST</acronym> table alone, so drop the original first when
+changing a table in place.
+</para>
+
+<para>
+A table that has no <acronym>TOAST</acronym> table yet takes no dump and
+restore, as the value recorded now is applied when a wide column added later
+causes one to be created.
+</para>
+
+<para>
+A table rewrite such as <command>CLUSTER</command>,
+<command>REPACK</command> or <command>VACUUM FULL</command> preserves the
+type already in use, and so does a <application>pg_upgrade</application>,
+which carries the existing <acronym>TOAST</acronym> table over to the new
+cluster. The only way to recreate the <acronym>TOAST</acronym> table of a
+table is to dump and restore it.
+</para>
+
+<para>
+The type in use by the <acronym>TOAST</acronym> table of a table can be
+checked with:
+<programlisting>
+SELECT a.atttypid::regtype
+ FROM pg_class c, pg_attribute a
+ WHERE c.oid = 'mytable'::regclass AND
+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';
+</programlisting>
+</para>
+
+<note>
+<para>
+To change the <structfield>chunk_id</structfield> type of a table back to
+OID, set <literal>toast_value_type</literal> to <literal>oid</literal> and
+then dump and restore the table the same way. The
+chunk identifiers are assigned afresh during the restore, so the ones that did
+not fit in an OID are replaced by ones that do, and no error or
+warning is reported. Such a table is however limited again to 2^32
+out-of-line values, and as the OID space fills up, finding an OID that is
+still free when inserting into the <acronym>TOAST</acronym> table can
+become expensive. The server log reports <quote>still searching for an unused
+OID in relation</quote> when that happens.
+</para>
+</note>
+</sect2>
+
</sect1>
<sect1 id="storage-fsm">
--
2.47.3
v1-0003-pg_dump-Preserve-the-chunk_id-type-of-a-TOAST-tab.patch (application/octet-stream, 9.2K) download
From 28acfb6db870802903fdb0fd1c41f24ea2872fd1 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 14 Sep 2026 18:43:33 +0000
Subject: [PATCH v1 3/3] pg_dump: Preserve the chunk_id type of a TOAST table.
Previously, pg_dump built the WITH clause of a relation from its
reloptions alone. As toast_value_type is read only when a TOAST
table is created, a reset of it removes the entry from the
reloptions but leaves the TOAST table alone, which keeps the OID8
chunk_id it was created with. pg_attribute is then the only place
recording the type in use, so the dump of such a relation carried
no type and the restore created its TOAST table with an OID
instead. Nothing reported an error, as the chunk identifiers are
assigned afresh by the restore, but the relation was limited to
2^32 out-of-line values again.
Fix this by reporting the type of chunk_id in use, read from
pg_attribute, when the reloptions do not mention it. A value
present in the reloptions is a request to change the type, and
takes precedence, so that setting it and then dumping and
restoring changes the type.
Note that nothing is reported in binary upgrade mode, where the
type is carried by binary_upgrade_set_next_toast_chunk_id_typoid
without reading the reloption, so pg_upgrade preserved the type
all along.
Add a test for this in the pg_upgrade TAP test.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/aqd_gYs35v--sVlR%40paquier.xyz
---
doc/src/sgml/ref/create_table.sgml | 4 +-
src/bin/pg_dump/pg_dump.c | 58 +++++++++++++++++++++++++-
src/bin/pg_dump/pg_dump.h | 2 +
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 16 +++++++
4 files changed, 78 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index fef24d8f3a2..c7456f0f788 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1678,7 +1678,9 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<link linkend="sql-vacuum"><command>VACUUM FULL</command></link>
preserves the type already in use. A dump and restore, on the other
hand, recreates the <acronym>TOAST</acronym> relation and therefore
- applies the current value of this parameter.
+ applies this parameter when it is set. When it is not set,
+ <application>pg_dump</application> reports the type used by the existing
+ <acronym>TOAST</acronym> table, so that a restore preserves it.
</para>
</listitem>
</varlistentry>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 388c3b9c346..f54cfcbba10 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -7154,6 +7154,7 @@ getTables(Archive *fout, int *numTables)
int i;
PQExpBuffer query = createPQExpBuffer();
TableInfo *tblinfo;
+ bool dump_toast_value_type;
int i_reltableoid;
int i_reloid;
int i_relname;
@@ -7187,6 +7188,7 @@ getTables(Archive *fout, int *numTables)
int i_reloptions;
int i_checkoption;
int i_toastreloptions;
+ int i_toastvaluetype;
int i_reloftype;
int i_foreignserver;
int i_amname;
@@ -7212,6 +7214,9 @@ getTables(Archive *fout, int *numTables)
* wrong answers if any concurrent DDL is happening.
*/
+ dump_toast_value_type = (fout->remoteVersion >= 200000 &&
+ !dopt->binary_upgrade);
+
appendPQExpBufferStr(query,
"SELECT c.tableoid, c.oid, c.relname, "
"c.relnamespace, c.relkind, c.reltype, "
@@ -7266,6 +7271,34 @@ getTables(Archive *fout, int *numTables)
"CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text "
"WHEN 'check_option=cascaded' = ANY (c.reloptions) THEN 'CASCADED'::text ELSE NULL END AS checkoption, ");
+ /*
+ * A reset of toast_value_type removes the entry from the reloptions of a
+ * relation but leaves its TOAST table alone, which keeps the chunk_id
+ * type it was created with. The reloptions then say nothing about the
+ * type in use, and a WITH clause built from the reloptions alone would
+ * have the restore create the TOAST table with the default type instead.
+ * Report the type of chunk_id for such a relation. Only OID8 needs to be
+ * reported, as a relation using OID matches the default. A value present
+ * in the reloptions is a request to change the type, and takes
+ * precedence.
+ *
+ * Nothing is reported in binary upgrade mode. There the type is carried
+ * by binary_upgrade_set_next_toast_chunk_id_typoid and the reloption is
+ * not read at all, so a WITH clause would only add an entry the
+ * reloptions of the old cluster do not have.
+ */
+ if (dump_toast_value_type)
+ appendPQExpBufferStr(query,
+ "CASE WHEN ta.atttypid = "
+ CppAsString2(OID8OID) " AND "
+ "NOT EXISTS (SELECT 1 FROM "
+ "unnest(coalesce(c.reloptions, '{}')) AS o "
+ "WHERE split_part(o, '=', 1) = 'toast_value_type') "
+ "THEN 'oid8' ELSE NULL END AS toast_value_type, ");
+ else
+ appendPQExpBufferStr(query,
+ "NULL AS toast_value_type, ");
+
appendPQExpBufferStr(query,
"am.amname, ");
@@ -7306,6 +7339,15 @@ getTables(Archive *fout, int *numTables)
" AND tc.relkind = " CppAsString2(RELKIND_TOASTVALUE)
" AND c.relkind <> " CppAsString2(RELKIND_PARTITIONED_TABLE) ")\n");
+ /*
+ * Left join to pg_attribute to pick up the type of chunk_id in use by the
+ * TOAST table.
+ */
+ if (dump_toast_value_type)
+ appendPQExpBufferStr(query,
+ "LEFT JOIN pg_attribute ta ON (ta.attrelid = tc.oid"
+ " AND ta.attname = 'chunk_id')\n");
+
/*
* Restrict to interesting relkinds (in particular, not indexes). Not all
* relkinds are possible in older servers, but it's not worth the trouble
@@ -7377,6 +7419,7 @@ getTables(Archive *fout, int *numTables)
i_reloptions = PQfnumber(res, "reloptions");
i_checkoption = PQfnumber(res, "checkoption");
i_toastreloptions = PQfnumber(res, "toast_reloptions");
+ i_toastvaluetype = PQfnumber(res, "toast_value_type");
i_reloftype = PQfnumber(res, "reloftype");
i_foreignserver = PQfnumber(res, "foreignserver");
i_amname = PQfnumber(res, "amname");
@@ -7458,6 +7501,10 @@ getTables(Archive *fout, int *numTables)
else
tblinfo[i].checkoption = pg_strdup(PQgetvalue(res, i, i_checkoption));
tblinfo[i].toast_reloptions = pg_strdup(PQgetvalue(res, i, i_toastreloptions));
+ if (PQgetisnull(res, i, i_toastvaluetype))
+ tblinfo[i].toast_value_type = NULL;
+ else
+ tblinfo[i].toast_value_type = pg_strdup(PQgetvalue(res, i, i_toastvaluetype));
tblinfo[i].reloftype = atooid(PQgetvalue(res, i, i_reloftype));
tblinfo[i].foreign_server = atooid(PQgetvalue(res, i, i_foreignserver));
if (PQgetisnull(res, i, i_amname))
@@ -17390,7 +17437,8 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
}
if (nonemptyReloptions(tbinfo->reloptions) ||
- nonemptyReloptions(tbinfo->toast_reloptions))
+ nonemptyReloptions(tbinfo->toast_reloptions) ||
+ tbinfo->toast_value_type != NULL)
{
bool addcomma = false;
@@ -17400,6 +17448,14 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
addcomma = true;
appendReloptionsArrayAH(q, tbinfo->reloptions, "", fout);
}
+ if (tbinfo->toast_value_type != NULL)
+ {
+ if (addcomma)
+ appendPQExpBufferStr(q, ", ");
+ addcomma = true;
+ appendPQExpBuffer(q, "toast_value_type=%s",
+ tbinfo->toast_value_type);
+ }
if (nonemptyReloptions(tbinfo->toast_reloptions))
{
if (addcomma)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 2bbb5d5773b..0a9089a815a 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -315,6 +315,8 @@ typedef struct _tableInfo
char *reloptions; /* options specified by WITH (...) */
char *checkoption; /* WITH CHECK OPTION, if any */
char *toast_reloptions; /* WITH options for the TOAST table */
+ char *toast_value_type; /* type of chunk_id in the TOAST table,
+ * when not implied by reloptions */
bool hasindex; /* does it have any indexes? */
bool hasrules; /* does it have any rules? */
bool hastriggers; /* does it have any triggers? */
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 9590625fc32..493a3dc4d8b 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -456,6 +456,22 @@ push(@dump_command, '--extra-float-digits', '0')
if ($oldnode->pg_version < 12);
$newnode->command_ok(\@dump_command, 'dump before running pg_upgrade');
+# A dump reports the type of chunk_id when the reloptions say nothing about
+# it, so that a restore recreates the TOAST table with the same type.
+# toasttest_oid8 is defined in strings.sql, which resets the reloption.
+if (!defined($ENV{olddump}))
+{
+ like(
+ slurp_file($dump1_file),
+ qr/^
+ \QCREATE TABLE public.toasttest_oid8 (\E\n
+ \s+\Qf1 bytea\E\n
+ \Q)\E\n
+ \QWITH (toast_value_type=oid8);\E
+ /xm,
+ 'dump reports the chunk_id type after a reset of toast_value_type');
+}
+
# After dumping, update references to the old source tree's regress.so
# to point to the new tree.
if (defined($ENV{oldinstall}))
--
2.47.3
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-21 22:53 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
1 sibling, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-21 22:53 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 21, 2026 at 03:39:23PM -0700, Bharath Rupireddy wrote:
> On Mon, Sep 21, 2026 at 4:11 AM John Naylor <johncnaylorls@gmail.com> wrote:
>> It looks like doc/src/sgml/limits.sgml needs to be updated as well.
>
> Nice catch. Thanks for pointing it out. Please find the attached patch for that.
Yeah, I've missed a spot that required a refresh.
> I also attached the 0001 and 0002 from
> https://www.postgresql.org/message-id/CALj2ACUmBhBGb%2B4d8SRq9JT\_ZsObCAvY6-10CKgSWX98zdv2DQ%40mail.gmail.com
> here as 0002 and 0003.
Not feeling much about 0002 at this stage. I don't disagree about the
fact of documenting something, but one has a few more ways to do it,
one being a CTAS with a WITH clause. You could also drop the varlena
attributes from a definition (after having migrated the values),
VACUUM FULL to drop the existing TOAST and add a new text attribute to
force the creation of a new TOAST table based on the new type wanted,
after an ALTER TABLE SET.
> In the attached 0003, I fixed an issue with the remote version check
> and simplified the tests by reusing an existing table whose reloption
> is already reset, and verifying that the dump reports the original
> TOAST type. This is good enough to cover the new code without starting
> a new instance just to test this. I also used a join instead of a
> per-relation lookup in the query to get the TOAST type, which avoids
> an extra catalog lookup for every table.
I still don't think much about this part, FWIW. That's just switching
some semantics to a different one which shows unclear benefits (aka it
is about what we should do on the dump side if we find a reloption set
or not, vs what's stored on disk).
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-22 00:07 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-22 00:07 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Mon, Sep 21, 2026 at 3:53 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> > I also attached the 0001 and 0002 from
> > https://www.postgresql.org/message-id/CALj2ACUmBhBGb%2B4d8SRq9JT\_ZsObCAvY6-10CKgSWX98zdv2DQ%40mail.gmail.com
> > here as 0002 and 0003.
>
> Not feeling much about 0002 at this stage. I don't disagree about the
> fact of documenting something, but one has a few more ways to do it,
> one being a CTAS with a WITH clause. You could also drop the varlena
> attributes from a definition (after having migrated the values),
> VACUUM FULL to drop the existing TOAST and add a new text attribute to
> force the creation of a new TOAST table based on the new type wanted,
> after an ALTER TABLE SET.
Agreed. Since we don't exactly know which option works better for
users, having one thing documented in core tends to be taken as the
recommended approach, which I would like to avoid. So, I'm fine to
withdraw this.
> > In the attached 0003, I fixed an issue with the remote version check
> > and simplified the tests by reusing an existing table whose reloption
> > is already reset, and verifying that the dump reports the original
> > TOAST type. This is good enough to cover the new code without starting
> > a new instance just to test this. I also used a join instead of a
> > per-relation lookup in the query to get the TOAST type, which avoids
> > an extra catalog lookup for every table.
>
> I still don't think much about this part, FWIW. That's just switching
> some semantics to a different one which shows unclear benefits (aka it
> is about what we should do on the dump side if we find a reloption set
> or not, vs what's stored on disk).
Agreed here as well. I expect one does not frequently reset the TOAST
reloption in production, especially after knowing that it doesn't
affect the table once created and has the data. If anyone raises this
later, at least there is evidence in this thread that it was
discussed. They can set the reloption to match the TOAST table
chunk_id type if they wish to. So, I'm fine to withdraw this as well.
Apologies for the noise. Thanks.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-22 07:57 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
1 sibling, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-22 07:57 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 21, 2026 at 03:39:23PM -0700, Bharath Rupireddy wrote:
> On Mon, Sep 21, 2026 at 4:11 AM John Naylor <johncnaylorls@gmail.com> wrote:
>> It looks like doc/src/sgml/limits.sgml needs to be updated as well.
>
> Nice catch. Thanks for pointing it out. Please find the attached patch for that.
- values are large enough to require it. Only an 18-byte pointer must remain
- inside the tuple in the table's heap. For shorter length variable-length
- fields, either a 4-byte or 1-byte field header is used and the value is
- stored inside the heap tuple.
+ values are large enough to require it. Only an 18-byte or 22-byte pointer
+ must remain inside the tuple in the table's heap, for a TOAST table using an
+ OID or an OID8 respectively. For shorter length variable-length fields,
+ either a 4-byte or 1-byte field header is used and the value is stored
+ inside the heap tuple.
How about the following instead for this part, slightly tweaked:
"Only an 18-byte or 22-byte TOAST pointer must remain inside the tuple
in the table's heap, depending on whether the TOAST table uses OID or
OID8 values, respectively. For shorter length variable-length fields,
either a 4-byte or 1-byte field header is used and the value is stored
inside the heap tuple."
Typically, this is only an issue for tables containing many terabytes
of data; partitioning is a possible workaround.
+ A TOAST table using an OID8 does not have this issue, as it can store up
+ to 2^64 out-of-line values and new identifiers are assigned without
+ searching for a free one; see <xref linkend="reloption-toast-value-type"/>.
And here, a bit cleaned up:
"A TOAST table using OID8 values does not have this issue as it can
store up to 2^64 out-of-line values. New identifiers are assigned
without searching for a free one.
See <xref linkend="reloption-toast-value-type"/> for details."
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-22 22:35 Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Bharath Rupireddy @ 2026-09-22 22:35 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Tue, Sep 22, 2026 at 12:58 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> - values are large enough to require it. Only an 18-byte pointer must remain
> - inside the tuple in the table's heap. For shorter length variable-length
> - fields, either a 4-byte or 1-byte field header is used and the value is
> - stored inside the heap tuple.
> + values are large enough to require it. Only an 18-byte or 22-byte pointer
> + must remain inside the tuple in the table's heap, for a TOAST table using an
> + OID or an OID8 respectively. For shorter length variable-length fields,
> + either a 4-byte or 1-byte field header is used and the value is stored
> + inside the heap tuple.
>
> How about the following instead for this part, slightly tweaked:
> "Only an 18-byte or 22-byte TOAST pointer must remain inside the tuple
> in the table's heap, depending on whether the TOAST table uses OID or
> OID8 values, respectively. For shorter length variable-length fields,
> either a 4-byte or 1-byte field header is used and the value is stored
> inside the heap tuple."
>
> Typically, this is only an issue for tables containing many terabytes
> of data; partitioning is a possible workaround.
> + A TOAST table using an OID8 does not have this issue, as it can store up
> + to 2^64 out-of-line values and new identifiers are assigned without
> + searching for a free one; see <xref linkend="reloption-toast-value-type"/>.
>
> And here, a bit cleaned up:
> "A TOAST table using OID8 values does not have this issue as it can
> store up to 2^64 out-of-line values. New identifiers are assigned
> without searching for a free one.
> See <xref linkend="reloption-toast-value-type"/> for details."
Agreed. Using OID/OID8 values and breaking up the sentences reads
better. Please find the attached v2 patch.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
Attachments:
v2-0001-doc-Mention-oid8-TOAST-values-in-the-limits-appen.patch (application/octet-stream, 2.4K) download
From d2b58dfb352f3f07b047867714552d6f3ebbc8c7 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Tue, 22 Sep 2026 22:13:30 +0000
Subject: [PATCH v2] doc: Mention oid8 TOAST values in the limits appendix
Mention the 22-byte on-disk TOAST pointer used when a TOAST table
stores oid8 values, next to the existing 18-byte one used for oid
values, and note that such a table can store up to 2^64
out-of-line values, with new identifiers assigned without
searching for a free one.
Oversight in 6e58d6356fcb.
Reported-by: John Naylor <johncnaylorls@gmail.com>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CANWCAZa+pWZL1Zjep1pz1F0sVWwNOQf+QfGErOaiUvC10Yr2XA@mail.gmail.com
---
doc/src/sgml/limits.sgml | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/limits.sgml b/doc/src/sgml/limits.sgml
index f26f4466719..6c8699b516f 100644
--- a/doc/src/sgml/limits.sgml
+++ b/doc/src/sgml/limits.sgml
@@ -123,10 +123,11 @@
Variable-length fields of
types such as <type>text</type>, <type>varchar</type>, and <type>char</type>
can have their values stored out of line in the table's TOAST table when the
- values are large enough to require it. Only an 18-byte pointer must remain
- inside the tuple in the table's heap. For shorter length variable-length
- fields, either a 4-byte or 1-byte field header is used and the value is
- stored inside the heap tuple.
+ values are large enough to require it. Only an 18-byte or 22-byte TOAST
+ pointer must remain inside the tuple in the table's heap, depending on
+ whether the TOAST table uses OID or OID8 values, respectively. For shorter
+ length variable-length fields, either a 4-byte or 1-byte field header is used
+ and the value is stored inside the heap tuple.
</para>
<para>
@@ -145,5 +146,8 @@
can become expensive, in turn slowing down INSERT/UPDATE statements.
Typically, this is only an issue for tables containing many terabytes
of data; partitioning is a possible workaround.
+ A TOAST table using OID8 values does not have this issue as it can store up
+ to 2^64 out-of-line values. New identifiers are assigned without searching
+ for a free one. See <xref linkend="reloption-toast-value-type"/> for details.
</para>
</appendix>
--
2.47.3
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-23 10:57 Michael Paquier <michael@paquier.xyz>
parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
0 siblings, 1 reply; 72+ messages in thread
From: Michael Paquier @ 2026-09-23 10:57 UTC (permalink / raw)
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Sep 22, 2026 at 03:35:00PM -0700, Bharath Rupireddy wrote:
> Agreed. Using OID/OID8 values and breaking up the sentences reads
> better. Please find the attached v2 patch.
Pushed.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-30 10:44 Hannu Krosing <hannuk@google.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 72+ messages in thread
From: Hannu Krosing @ 2026-09-30 10:44 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
In my insert tests 8-byte TOAST was much faster that 4-byte toast
During a 30 min run inserting tuples with 8 toasted fields of ~100
bytes oid8 toast kept a steady 18k inserts / second against 14.6k
inserts second for oid4
This resulted in a solid 23% speed-up because it didn't need to verify
new OIDs against the unique index, despite each toast index entry
taking up slightly more space (30.1 bytes vs. 21.4 bytes on average)
On Wed, Sep 23, 2026 at 12:58 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Tue, Sep 22, 2026 at 03:35:00PM -0700, Bharath Rupireddy wrote:
> > Agreed. Using OID/OID8 values and breaking up the sentences reads
> > better. Please find the attached v2 patch.
>
> Pushed.
> --
> Michael
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: Support for 8-byte TOAST values, round two
@ 2026-09-30 23:22 Michael Paquier <michael@paquier.xyz>
parent: Hannu Krosing <hannuk@google.com>
0 siblings, 0 replies; 72+ messages in thread
From: Michael Paquier @ 2026-09-30 23:22 UTC (permalink / raw)
To: Hannu Krosing <hannuk@google.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; John Naylor <johncnaylorls@gmail.com>; Greg Burd <greg@burd.me>; Yugo Nagata <nagata@sraoss.co.jp>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Sep 30, 2026 at 12:44:28PM +0200, Hannu Krosing wrote:
> In my insert tests 8-byte TOAST was much faster that 4-byte toast
>
> This resulted in a solid 23% speed-up because it didn't need to verify
> new OIDs against the unique index, despite each toast index entry
> taking up slightly more space (30.1 bytes vs. 21.4 bytes on average)
The speedup is not surprising. Thanks for posting some numbers.
A note about that. Postgres hackers had a meetup two weeks ago in
Tokyo where I have reported the activity of this thread, including the
fact that the new values are not rechecked in the INSERT path.
My opinion, same as upthread, is that a recheck is kind of pointless
because we should never go backwards under normal running conditions
and we would run out of LSNs before we reach the OID8 limit. I did
mention the argument of using pg_resetwal -o to go backwards, leading
to INSERT failures due to duplicates of the TOAST unique index.
Fujii-san had a more biased opinion than mine, mentioning that a
recheck could be a more defensive position, even if it costs in normal
running conditions.
Perhaps my opinion will be overruled in this release cycle, which is
fine, I just want to point out that the API toastid_valueid_exists()
is transparently able to work with a recheck of Oid8 values, so the
value recheck could be plugged in for this case as well, depending on
how the discussion flows. Perhaps there is a bug I am not aware of
yet, that could justify the recheck, of course.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 72+ messages in thread
end of thread, other threads:[~2026-09-30 23:22 UTC | newest]
Thread overview: 72+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-05-08 06:07 Support for 8-byte TOAST values, round two Michael Paquier <michael@paquier.xyz>
2026-07-30 04:38 ` Yugo Nagata <nagata@sraoss.co.jp>
2026-07-30 08:01 ` Yugo Nagata <nagata@sraoss.co.jp>
2026-07-30 11:16 ` Hannu Krosing <hannuk@google.com>
2026-07-31 02:36 ` Michael Paquier <michael@paquier.xyz>
2026-07-31 10:58 ` Hannu Krosing <hannuk@google.com>
2026-07-31 11:10 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2026-07-31 14:16 ` Hannu Krosing <hannuk@google.com>
2026-08-06 04:38 ` Michael Paquier <michael@paquier.xyz>
2026-08-09 21:53 ` Hannu Krosing <hannuk@google.com>
2026-08-10 00:08 ` Michael Paquier <michael@paquier.xyz>
2026-08-10 07:37 ` Hannu Krosing <hannuk@google.com>
2026-08-10 08:03 ` Michael Paquier <michael@paquier.xyz>
2026-08-10 10:43 ` Hannu Krosing <hannuk@google.com>
2026-08-13 06:38 ` Michael Paquier <michael@paquier.xyz>
2026-09-02 20:27 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-03 04:58 ` Michael Paquier <michael@paquier.xyz>
2026-09-04 04:28 ` Michael Paquier <michael@paquier.xyz>
2026-09-05 03:11 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-07 07:54 ` Michael Paquier <michael@paquier.xyz>
2026-09-08 03:50 ` Michael Paquier <michael@paquier.xyz>
2026-09-08 23:55 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-09 00:58 ` Michael Paquier <michael@paquier.xyz>
2026-09-09 14:14 ` Greg Burd <greg@burd.me>
2026-09-09 17:14 ` Greg Burd <greg@burd.me>
2026-09-09 23:18 ` Michael Paquier <michael@paquier.xyz>
2026-09-10 07:51 ` Michael Paquier <michael@paquier.xyz>
2026-09-10 17:06 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-10 21:14 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-11 07:33 ` Michael Paquier <michael@paquier.xyz>
2026-09-11 15:44 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-11 17:07 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-14 00:46 ` Michael Paquier <michael@paquier.xyz>
2026-09-14 05:00 ` Michael Paquier <michael@paquier.xyz>
2026-09-14 14:01 ` Greg Burd <greg@burd.me>
2026-09-14 19:26 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-15 07:08 ` Michael Paquier <michael@paquier.xyz>
2026-09-15 15:26 ` Greg Burd <greg@burd.me>
2026-09-21 11:11 ` John Naylor <johncnaylorls@gmail.com>
2026-09-21 22:39 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-21 22:53 ` Michael Paquier <michael@paquier.xyz>
2026-09-22 00:07 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-22 07:57 ` Michael Paquier <michael@paquier.xyz>
2026-09-22 22:35 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-23 10:57 ` Michael Paquier <michael@paquier.xyz>
2026-09-30 10:44 ` Hannu Krosing <hannuk@google.com>
2026-09-30 23:22 ` Michael Paquier <michael@paquier.xyz>
2026-09-14 23:43 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-15 07:59 ` Michael Paquier <michael@paquier.xyz>
2026-09-15 16:20 ` Rui Zhao <zhaorui126@gmail.com>
2026-09-15 17:05 ` Greg Burd <greg@burd.me>
2026-09-16 00:34 ` Michael Paquier <michael@paquier.xyz>
2026-09-16 00:29 ` Michael Paquier <michael@paquier.xyz>
2026-09-15 22:22 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-16 01:59 ` Michael Paquier <michael@paquier.xyz>
2026-09-16 17:48 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-16 23:50 ` Michael Paquier <michael@paquier.xyz>
2026-09-17 19:03 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-19 00:03 ` Michael Paquier <michael@paquier.xyz>
2026-09-19 00:26 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-11 16:03 ` Marcos Pegoraro <marcos@f10.com.br>
2026-09-14 05:12 ` Michael Paquier <michael@paquier.xyz>
2026-09-11 17:38 ` Rui Zhao <zhaorui126@gmail.com>
2026-09-11 23:30 ` Michael Paquier <michael@paquier.xyz>
2026-09-12 17:46 ` Rui Zhao <zhaorui126@gmail.com>
2026-09-11 03:42 ` Michael Paquier <michael@paquier.xyz>
2026-09-09 17:13 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-13 16:22 ` Oleg Bartunov <obartunov@postgrespro.ru>
2026-09-13 23:17 ` Michael Paquier <michael@paquier.xyz>
2026-09-13 23:51 ` Michael Paquier <michael@paquier.xyz>
2026-09-14 01:39 ` Oleg Bartunov <obartunov@postgrespro.ru>
2026-09-14 02:16 ` Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox