From: Michael Paquier <michael@paquier.xyz>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Yugo Nagata <nagata@sraoss.co.jp>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Support for 8-byte TOAST values, round two
Date: Tue, 8 Sep 2026 12:50:11 +0900
Message-ID: <ap-F89TpWEq324ij@paquier.xyz> (raw)
In-Reply-To: <ap5tu4VkJgsEthVq@paquier.xyz>
References: <af19kUjwjhaoUTLn@paquier.xyz>
<20260730133818.a775f48723ffea5118794b2a@sraoss.co.jp>
<20260730170102.15b4aded1698dee3c2a304e2@sraoss.co.jp>
<anQP1CWmZHGsnpwU@paquier.xyz>
<CALj2ACVnO6jjs5aRx3dSOBy9LV7=iakPBR1eOvoeB8rL8Q47zg@mail.gmail.com>
<apj-a9ALfyMOGSCF@paquier.xyz>
<CALj2ACWmEafrjiZSekxAM+eVWtkX5uQ0B6fYjFbcfZ23fiSk5A@mail.gmail.com>
<ap5tu4VkJgsEthVq@paquier.xyz>
On Mon, Sep 07, 2026 at 04:54:35PM +0900, Michael Paquier wrote:
> Attached is v14, with all that addressed (should perhaps have moved
> the tests in 0008 to 0007, but it's late here and that does not change
> the stuff to review).
While doing a new round of self-review, that included some tests with
pg_upgrade, I have spotted a hole in the control file: the nextOid was
stored as an Oid instead of an Oid8. This would be bad when
upgrading, because we would conflict for oid8 TOAST values once the
counter gets higher than 2^32. This has also spotted a gap in
pg_resetwal and pg_control_checkpoint(), where we need to show the new
information as an oid8.
Thinking more about which part of the patch set makes the most sense
as an independent piece, I have decided to split the upgrade of
nextOid to Oid8 in the control file and the introduction of the new
GetNewObjectId8() as a patch of their own, labelled now at 0001. This
includes a battery of tests for pg_upgrade to check that the value is
correctly carried, resetwal and pg_control_checkpoint(). As a whole,
I'm feeling pretty much OK with how this part bundles.
Small note: while hacking on 0001 I have managed to break pg_resetwal
-o with negative values. Added a test for that on HEAD separately,
while on it..
There is still something that I am wondering about this part of the
patch, though, should we still publish in the output of
pg_control_checkpoint() and pg_controldata an Oid extracted from the
Oid8 value? I have not done that, and that does not make the control
file larger. Just wondering if that would be useful to keep because
we cannot really do an oid8->oid cast, and the oid information could
still be useful on its own?
The rest of the patch set is mostly following the same organization,
and I have spent a few hours working more on the last piece, making
cleaner the contract with the vartag handling (one could not the
changes in varatt.h, where I have changed to inline function for the
get/set Oid8 values in varatt_external_oid8).
0005~0007 in the set of patches cover gaps in the regression tests.
0005 is the same one set of SQLs as the previous patch set (mostly),
0006 and 0007 cover more cases with test_decoding, amcheck,
reloptions, the compression function, and the external toast pointer
comparison in toast_tuple_init(). They have been split as they can
still work once the reloption supports oid8, even if we don't make the
varatt_external added yet (last patch).
--
Michael
Attachments:download all (12 files, .tar.gz)v15-0001-Enlarge-OID-generation-to-8-bytes.patch (text/plain, 23.3K) download
From 7e71e23982b4fe1ff105b4e7f7731b5c10bdb0af Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 12:44:07 +0900
Subject: [PATCH v15 01/11] Enlarge OID generation to 8 bytes
This change is combined with an update of the control file, with
CheckPoint.nextOid enlarged from Oid to Oid8. pg_control_checkpoint(),
pg_resetwal and pg_upgrade are updated to accomodate to this change, to
make sure that the value is not truncated due to 32-bit limitations when
used.
This commit adds a new routine called GetNewObjectId8() in varsup.c,
which is able to retrieve a 8-byte OID. GetNewObjectId() is kept
compatible with its origin, where we still check that the lower 32 bits
of the counter do not wraparound, handling the FirstNormalObjectId case.
A couple of comments in the xlog redo logic referred to wraparound
issues, which is not the case anymore with this change in place, so
refresh these.
Tests are added to cover that, across all the tools covered by this
change.
Bump catalog version.
Bump XLOG_PAGE_MAGIC, due to XLOG_NEXTOID.
---
src/include/access/transam.h | 3 +-
src/include/access/xlog.h | 2 +-
src/include/access/xlog_internal.h | 2 +-
src/include/catalog/pg_control.h | 4 +-
src/include/catalog/pg_proc.dat | 2 +-
src/backend/access/rmgrdesc/xlogdesc.c | 8 +--
src/backend/access/transam/varsup.c | 59 ++++++++++++++++-------
src/backend/access/transam/xlog.c | 27 ++++++-----
src/backend/access/transam/xlogrecovery.c | 2 +-
src/backend/utils/misc/pg_controldata.c | 2 +-
src/bin/pg_controldata/pg_controldata.c | 2 +-
src/bin/pg_resetwal/pg_resetwal.c | 10 ++--
src/bin/pg_resetwal/t/001_basic.pl | 17 +++++++
src/bin/pg_upgrade/controldata.c | 2 +-
src/bin/pg_upgrade/pg_upgrade.c | 2 +-
src/bin/pg_upgrade/pg_upgrade.h | 2 +-
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 25 ++++++++++
doc/src/sgml/func/func-info.sgml | 2 +-
doc/src/sgml/ref/pg_resetwal.sgml | 6 +--
19 files changed, 123 insertions(+), 56 deletions(-)
diff --git a/src/include/access/transam.h b/src/include/access/transam.hindex 55a4ab26b348..f687df1edc60 100644--- a/src/include/access/transam.h+++ b/src/include/access/transam.h@@ -211,7 +211,7 @@ typedef struct TransamVariablesData
/*
* These fields are protected by OidGenLock.
*/
- Oid nextOid; /* next OID to assign */+ Oid8 nextOid; /* next OID (8 bytes) to assign */
uint32 oidCount; /* OIDs available before must do XLOG work */
/*
@@ -353,6 +353,7 @@ extern void SetTransactionIdLimit(TransactionId oldest_datfrozenxid,
extern void AdvanceOldestClogXid(TransactionId oldest_datfrozenxid);
extern bool ForceTransactionIdLimitUpdate(void);
extern Oid GetNewObjectId(void);
+extern Oid8 GetNewObjectId8(void);
extern void StopGeneratingPinnedObjectIds(void);
#ifdef USE_ASSERT_CHECKING
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.hindex 130ba9291096..7a590b7e1eaa 100644--- a/src/include/access/xlog.h+++ b/src/include/access/xlog.h@@ -284,7 +284,7 @@ extern void ShutdownXLOG(int code, Datum arg);
extern bool CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
extern WALAvailability GetWALAvailability(XLogRecPtr targetLSN);
-extern void XLogPutNextOid(Oid nextOid);+extern void XLogPutNextOid(Oid8 nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern XLogRecPtr XLogAssignLSN(void);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.hindex be718993401b..bf609c3c703d 100644--- a/src/include/access/xlog_internal.h+++ b/src/include/access/xlog_internal.h@@ -32,7 +32,7 @@
/*
* Each page of XLOG file has a header like this:
*/
-#define XLOG_PAGE_MAGIC 0xD121 /* can be used as WAL version indicator */+#define XLOG_PAGE_MAGIC 0xD122 /* can be used as WAL version indicator */
typedef struct XLogPageHeaderData
{
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.hindex 7b5404460ec2..89ab43dd4fc7 100644--- a/src/include/catalog/pg_control.h+++ b/src/include/catalog/pg_control.h@@ -22,7 +22,7 @@
/* Version identifier for this pg_control format */
-#define PG_CONTROL_VERSION 1903+#define PG_CONTROL_VERSION 2000
/* Nonce key length, see below */
#define MOCK_AUTH_NONCE_LEN 32
@@ -43,7 +43,7 @@ typedef struct CheckPoint
int wal_level; /* current wal_level */
bool logicalDecodingEnabled; /* current logical decoding status */
FullTransactionId nextXid; /* next free transaction ID */
- Oid nextOid; /* next free OID */+ Oid8 nextOid; /* next free OID */
MultiXactId nextMulti; /* next free MultiXactId */
MultiXactOffset nextMultiOffset; /* next free MultiXact offset */
TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 960763ee50ba..c53ce68c7173 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -12429,7 +12429,7 @@
descr => 'pg_controldata checkpoint state information as a function',
proname => 'pg_control_checkpoint', provolatile => 'v',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,bool,text,oid,xid,int8,xid,oid,xid,xid,oid,xid,xid,int4,timestamptz}',+ proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,bool,text,oid8,xid,int8,xid,oid,xid,xid,oid,xid,xid,int4,timestamptz}',
proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,logical_decoding,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,data_page_checksum_version,checkpoint_time}',
prosrc => 'pg_control_checkpoint' },
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.cindex 2468a7d25785..64e749c1e6b7 100644--- a/src/backend/access/rmgrdesc/xlogdesc.c+++ b/src/backend/access/rmgrdesc/xlogdesc.c@@ -101,7 +101,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
CheckPoint *checkpoint = (CheckPoint *) rec;
appendStringInfo(buf, "redo %X/%08X; "
- "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "+ "tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid " OID8_FORMAT "; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
@@ -129,10 +129,10 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
}
else if (info == XLOG_NEXTOID)
{
- Oid nextOid;+ Oid8 nextOid;- memcpy(&nextOid, rec, sizeof(Oid));- appendStringInfo(buf, "%u", nextOid);+ memcpy(&nextOid, rec, sizeof(Oid8));+ appendStringInfo(buf, OID8_FORMAT, nextOid);
}
else if (info == XLOG_RESTORE_POINT)
{
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.cindex cb68d8fa974a..889bdebe1880 100644--- a/src/backend/access/transam/varsup.c+++ b/src/backend/access/transam/varsup.c@@ -541,31 +541,51 @@ ForceTransactionIdLimitUpdate(void)
/*
- * GetNewObjectId -- allocate a new OID+ * GetNewObjectId -- allocate a new OID (4 bytes)
*
- * OIDs are generated by a cluster-wide counter. Since they are only 32 bits- * wide, counter wraparound will occur eventually, and therefore it is unwise- * to assume they are unique unless precautions are taken to make them so.- * Hence, this routine should generally not be used directly. The only direct- * callers should be GetNewOidWithIndex() and GetNewRelFileNumber() in- * catalog/catalog.c.+ * OIDs are generated by a cluster-wide counter. The callers of this routine+ * expect a 32 bit-wide counter. Counter wraparound will occur eventually,+ * and therefore it is unwise to assume they are unique unless precautions are+ * taken to make them so. This routine should generally not be used directly.+ * The only direct callers should be GetNewOidWithIndex() and+ * GetNewRelFileNumber() in catalog/catalog.c.
*/
Oid
GetNewObjectId(void)
{
- Oid result;+ return (Oid) GetNewObjectId8();+}++/*+ * GetNewObjectId8 -- allocate a new OID (8 bytes)+ *+ * This routine can be called directly if the consumer of the OID allocated+ * stores the counter in an 8-byte space, where wraparound does not matter.+ * We still need to care about the wraparound case in the low 32 bits of the+ * space allocated, GetNewObjectId() expecting OIDs to never be allocated+ * up to FirstNormalObjectId.+ */+Oid8+GetNewObjectId8(void)+{+ Oid8 result;+ Oid nextoid_lo;+ uint32 nextoid_hi;
/* safety check, we should never get this far in a HS standby */
if (RecoveryInProgress())
elog(ERROR, "cannot assign OIDs during recovery");
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
+ nextoid_lo = (Oid) TransamVariables->nextOid;+ nextoid_hi = (uint32) (TransamVariables->nextOid >> 32);
/*
- * Check for wraparound of the OID counter. We *must* not return 0- * (InvalidOid), and in normal operation we mustn't return anything below- * FirstNormalObjectId since that range is reserved for initdb (see- * IsCatalogRelationOid()). Note we are relying on unsigned comparison.+ * Check for wraparound of the OID counter in its lower 4 bytes. We *must*+ * not return 0 (InvalidOid), and in normal operation we mustn't return+ * anything below FirstNormalObjectId since that range is reserved for+ * initdb (see IsCatalogRelationOid()). Note we are relying on unsigned+ * comparison.
*
* During initdb, we start the OID generator at FirstGenbkiObjectId, so we
* only wrap if before that point when in bootstrap or standalone mode.
@@ -575,26 +595,29 @@ GetNewObjectId(void)
* available for automatic assignment during initdb, while ensuring they
* will never conflict with user-assigned OIDs.
*/
- if (TransamVariables->nextOid < ((Oid) FirstNormalObjectId))+ if (nextoid_lo < ((Oid) FirstNormalObjectId))
{
if (IsPostmasterEnvironment)
{
/* wraparound, or first post-initdb assignment, in normal mode */
- TransamVariables->nextOid = FirstNormalObjectId;+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
else
{
/* we may be bootstrapping, so don't enforce the full range */
- if (TransamVariables->nextOid < ((Oid) FirstGenbkiObjectId))+ if (nextoid_lo < ((Oid) FirstGenbkiObjectId))
{
/* wraparound in standalone mode (unlikely but possible) */
- TransamVariables->nextOid = FirstNormalObjectId;+ nextoid_lo = FirstNormalObjectId;
TransamVariables->oidCount = 0;
}
}
}
+ /* Set next OID in its 8-byte space */+ TransamVariables->nextOid = ((Oid8) nextoid_hi) << 32 | nextoid_lo;+
/* If we run out of logged for use oids then we must log more */
if (TransamVariables->oidCount == 0)
{
@@ -619,7 +642,7 @@ GetNewObjectId(void)
* to the specified value.
*/
static void
-SetNextObjectId(Oid nextOid)+SetNextObjectId(Oid8 nextOid)
{
/* Safety check, this is only allowable during initdb */
if (IsPostmasterEnvironment)
@@ -629,7 +652,7 @@ SetNextObjectId(Oid nextOid)
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
if (TransamVariables->nextOid > nextOid)
- elog(ERROR, "too late to advance OID counter to %u, it is now %u",+ elog(ERROR, "too late to advance OID counter to " OID8_FORMAT ", it is now " OID8_FORMAT,
nextOid, TransamVariables->nextOid);
TransamVariables->nextOid = nextOid;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 3203f2fd4ee2..b84eac793ce8 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -8604,10 +8604,10 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
* Write a NEXTOID log record
*/
void
-XLogPutNextOid(Oid nextOid)+XLogPutNextOid(Oid8 nextOid)
{
XLogBeginInsert();
- XLogRegisterData(&nextOid, sizeof(Oid));+ XLogRegisterData(&nextOid, sizeof(Oid8));
(void) XLogInsert(RM_XLOG_ID, XLOG_NEXTOID);
/*
@@ -8873,16 +8873,17 @@ xlog_redo(XLogReaderState *record)
if (info == XLOG_NEXTOID)
{
- Oid nextOid;+ Oid8 nextOid;
/*
* We used to try to take the maximum of TransamVariables->nextOid and
- * the recorded nextOid, but that fails if the OID counter wraps- * around. Since no OID allocation should be happening during replay- * anyway, better to just believe the record exactly. We still take- * OidGenLock while setting the variable, just in case.+ * the recorded nextOid, but that failed back when the counter was 4+ * bytes wide and could wrap around. Since no OID allocation should+ * be happening during replay anyway, better to just believe the+ * record exactly. We still take OidGenLock while setting the+ * variable, just in case.
*/
- memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid));+ memcpy(&nextOid, XLogRecGetData(record), sizeof(Oid8));
LWLockAcquire(OidGenLock, LW_EXCLUSIVE);
TransamVariables->nextOid = nextOid;
TransamVariables->oidCount = 0;
@@ -9010,11 +9011,11 @@ xlog_redo(XLogReaderState *record)
* to track OID assignment through XLOG_NEXTOID records. The nextOid
* counter is from the start of the checkpoint and might well be stale
* compared to later XLOG_NEXTOID records. We could try to take the
- * maximum of the nextOid counter and our latest value, but since- * there's no particular guarantee about the speed with which the OID- * counter wraps around, that's a risky thing to do. In any case,- * users of the nextOid counter are required to avoid assignment of- * duplicates, so that a somewhat out-of-date value should be safe.+ * maximum of the nextOid counter and our latest value, but due to the+ * guarantees offered by XLogPutNextOid(), this is not necessary. In+ * any case, users of the nextOid counter are required to avoid+ * assignment of duplicates, so that a somewhat out-of-date value+ * should be safe.
*/
/* Handle multixact */
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.cindex acac97e89d38..fff8d57ac61d 100644--- a/src/backend/access/transam/xlogrecovery.c+++ b/src/backend/access/transam/xlogrecovery.c@@ -840,7 +840,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
LSN_FORMAT_ARGS(checkPoint.redo),
wasShutdown ? "true" : "false"));
ereport(DEBUG1,
- (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: %u",+ (errmsg_internal("next transaction ID: " UINT64_FORMAT "; next OID: " OID8_FORMAT,
U64FromFullTransactionId(checkPoint.nextXid),
checkPoint.nextOid)));
ereport(DEBUG1,
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.cindex 9014f0953e99..1f47302fe2a7 100644--- a/src/backend/utils/misc/pg_controldata.c+++ b/src/backend/utils/misc/pg_controldata.c@@ -124,7 +124,7 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid)));
nulls[7] = false;
- values[8] = ObjectIdGetDatum(ControlFile->checkPointCopy.nextOid);+ values[8] = ObjectId8GetDatum(ControlFile->checkPointCopy.nextOid);
nulls[8] = false;
values[9] = TransactionIdGetDatum(ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.cindex 6fc87ed114d8..b785f7f40701 100644--- a/src/bin/pg_controldata/pg_controldata.c+++ b/src/bin/pg_controldata/pg_controldata.c@@ -269,7 +269,7 @@ main(int argc, char *argv[])
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile->checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile->checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile->checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile->checkPointCopy.nextMulti);
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.cindex 79f3085d7696..63e4381e03f0 100644--- a/src/bin/pg_resetwal/pg_resetwal.c+++ b/src/bin/pg_resetwal/pg_resetwal.c@@ -82,7 +82,7 @@ static TransactionId oldest_commit_ts_xid_val;
static TransactionId newest_commit_ts_xid_val;
static bool next_oid_given = false;
-static Oid next_oid_val;+static Oid8 next_oid_val;
static bool mxids_given = false;
static MultiXactId next_mxid_val;
@@ -253,7 +253,7 @@ main(int argc, char *argv[])
case 'o':
errno = 0;
- next_oid_val = strtouint32_strict(optarg, &endptr, 0);+ next_oid_val = strtouint64_strict(optarg, &endptr, 0);
if (endptr == optarg || *endptr != '\0' || errno != 0)
{
pg_log_error("invalid argument for option %s", "-o");
@@ -775,7 +775,7 @@ PrintControlValues(bool guessed)
printf(_("Latest checkpoint's NextXID: %u:%u\n"),
EpochFromFullTransactionId(ControlFile.checkPointCopy.nextXid),
XidFromFullTransactionId(ControlFile.checkPointCopy.nextXid));
- printf(_("Latest checkpoint's NextOID: %u\n"),+ printf(_("Latest checkpoint's NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
printf(_("Latest checkpoint's NextMultiXactId: %u\n"),
ControlFile.checkPointCopy.nextMulti);
@@ -861,7 +861,7 @@ PrintNewControlValues(void)
if (next_oid_given)
{
- printf(_("NextOID: %u\n"),+ printf(_("NextOID: " OID8_FORMAT "\n"),
ControlFile.checkPointCopy.nextOid);
}
@@ -1236,7 +1236,7 @@ usage(void)
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
- printf(_(" -o, --next-oid=OID set next OID\n"));+ printf(_(" -o, --next-oid=OID8 set next OID (8 bytes)\n"));
printf(_(" -O, --multixact-offset=OFFSET set next multitransaction offset\n"));
printf(_(" -u, --oldest-transaction-id=XID set oldest transaction ID\n"));
printf(_(" -x, --next-transaction-id=XID set next transaction ID\n"));
diff --git a/src/bin/pg_resetwal/t/001_basic.pl b/src/bin/pg_resetwal/t/001_basic.plindex 6af97c0a7d99..f54e9bb42231 100644--- a/src/bin/pg_resetwal/t/001_basic.pl+++ b/src/bin/pg_resetwal/t/001_basic.pl@@ -265,4 +265,21 @@ command_like(
$node->start;
ok(1, 'server started after reset');
+# Checks for 8 bytes OIDs.+$node->stop;+my $new_nextoid = "4295067296"; # 2^32 + 100,000+command_ok([ 'pg_resetwal', '--next-oid' => $new_nextoid, $node->data_dir ],+ 'runs with --next-oid value above 2^32');+command_like(+ [ 'pg_resetwal', '--dry-run', $node->data_dir ],+ qr/^Latest checkpoint's NextOID: *$new_nextoid$/m,+ '8-byte --next-oid value preserved in pg_control');+$node->start;+is( $node->safe_psql(+ 'postgres',+ "SELECT next_oid >= '$new_nextoid'::oid8 FROM pg_control_checkpoint()"+ ),+ 't',+ 'new 8-byte OID reported');+
done_testing();
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.cindex 7543a9880451..f0fa2b1689fa 100644--- a/src/bin/pg_upgrade/controldata.c+++ b/src/bin/pg_upgrade/controldata.c@@ -245,7 +245,7 @@ get_control_data(ClusterInfo *cluster)
pg_fatal("%d: controldata retrieval problem", __LINE__);
p++; /* remove ':' char */
- cluster->controldata.chkpnt_nxtoid = str2uint(p);+ cluster->controldata.chkpnt_nxtoid = str2uint64(p);
got_oid = true;
}
else if ((p = strstr(bufin, "Latest checkpoint's NextMultiXactId:")) != NULL)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.cindex 7366fd4627cc..3704c620603c 100644--- a/src/bin/pg_upgrade/pg_upgrade.c+++ b/src/bin/pg_upgrade/pg_upgrade.c@@ -196,7 +196,7 @@ main(int argc, char **argv)
*/
prep_status("Setting next OID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
- "\"%s/pg_resetwal\" -o %u \"%s\"",+ "\"%s/pg_resetwal\" -o " OID8_FORMAT " \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtoid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.hindex c80e8fb4031c..4cb79bb780e6 100644--- a/src/bin/pg_upgrade/pg_upgrade.h+++ b/src/bin/pg_upgrade/pg_upgrade.h@@ -216,7 +216,7 @@ typedef struct
char nextxlogfile[25];
uint32 chkpnt_nxtxid;
uint32 chkpnt_nxtepoch;
- uint32 chkpnt_nxtoid;+ Oid8 chkpnt_nxtoid;
uint32 chkpnt_nxtmulti;
uint64 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.plindex a5cfd53265d4..728fd6b0b120 100644--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl@@ -588,6 +588,20 @@ $oldnode->start;
$oldnode->safe_psql('postgres', 'DROP DATABASE regression_invalid');
$oldnode->stop;
+# The OID counter is 8 bytes wide, check that it is carried. Older versions+# may not support 8-byte OIDs, so skip in this case.+my $big_next_oid = '4295067296'; # 2^32 + 100000+if (!defined($ENV{oldinstall}))+{+ command_ok(+ [+ 'pg_resetwal',+ '--next-oid' => $big_next_oid,+ $oldnode->data_dir+ ],+ 'set an 8-byte OID counter in the old instance');+}+
# --check command works here, cleans up pg_upgrade_output.d.
command_ok(
[
@@ -624,6 +638,17 @@ ok( !-d $newnode->data_dir . "/pg_upgrade_output.d",
$newnode->start;
+# The 8-byte OID has been carried.+if (!defined($ENV{oldinstall}))+{+ is( $newnode->safe_psql(+ 'postgres',+ "SELECT next_oid >= '$big_next_oid'::oid8 FROM pg_control_checkpoint()"+ ),+ 't',+ '8-byte OID counter is carried over by pg_upgrade');+}+
# Check if there are any logs coming from pg_upgrade, that would only be
# retained on failure.
my $log_path = $newnode->data_dir . "/pg_upgrade_output.d";
diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgmlindex 1ff023b741c8..e56c9a22c42a 100644--- a/doc/src/sgml/func/func-info.sgml+++ b/doc/src/sgml/func/func-info.sgml@@ -3433,7 +3433,7 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
<row>
<entry><structfield>next_oid</structfield></entry>
- <entry><type>oid</type></entry>+ <entry><type>oid8</type></entry>
</row>
<row>
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgmlindex 664c525f481c..a9580f35de95 100644--- a/doc/src/sgml/ref/pg_resetwal.sgml+++ b/doc/src/sgml/ref/pg_resetwal.sgml@@ -286,11 +286,11 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>-o <replaceable class="parameter">oid</replaceable></option></term>- <term><option>--next-oid=<replaceable class="parameter">oid</replaceable></option></term>+ <term><option>-o <replaceable class="parameter">oid8</replaceable></option></term>+ <term><option>--next-oid=<replaceable class="parameter">oid8</replaceable></option></term>
<listitem>
<para>
- Manually set the next OID.+ Manually set the next OID (8 bytes).
</para>
<para>
--
2.55.0
v15-0002-Refactor-some-TOAST-value-ID-code-to-use-Oid8-in.patch (text/plain, 16.1K) download
From 2a5e6c3fc4f27e5bfa417bf189e1154646af56a0 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 13:40:41 +0900
Subject: [PATCH v15 02/11] Refactor some TOAST value ID code to use Oid8
instead of Oid
This change is a mechanical switch to change most of the code paths that
assume TOAST value IDs to be Oids to become Oid8, easing an upcoming
change to allow larger TOAST values, at 8 bytes.
The areas touched are related to table AM, amcheck and logical
decoding's reorder buffer. A good chunk of the changes involve
switching printf() markers from %u to OID8_FORMAT.
---
src/include/access/heaptoast.h | 2 +-
src/include/access/tableam.h | 4 +-
src/backend/access/common/toast_internals.c | 8 +--
src/backend/access/heap/heaptoast.c | 12 ++--
.../replication/logical/reorderbuffer.c | 23 +++++---
contrib/amcheck/verify_heapam.c | 55 +++++++++++--------
6 files changed, 60 insertions(+), 44 deletions(-)
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.hindex b2346fd75bd4..aaae6e0fef69 100644--- a/src/include/access/heaptoast.h+++ b/src/include/access/heaptoast.h@@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* Fetch a slice from a toast value stored in a heap table.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,+extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.hindex ff03a2b816fc..ebec31a22e40 100644--- a/src/include/access/tableam.h+++ b/src/include/access/tableam.h@@ -783,7 +783,7 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
@@ -1987,7 +1987,7 @@ table_relation_toast_am(Relation rel)
* stored.
*/
static inline void
-table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,+table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid,
int32 attrsize, int32 sliceoffset,
int32 slicelength, varlena *result)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex 0f234661bfa9..ca5ed8dfabca 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -26,8 +26,8 @@
#include "utils/rel.h"
#include "utils/snapmgr.h"
-static bool toastrel_valueid_exists(Relation toastrel, Oid valueid);-static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);+static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);+static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
/* ----------
* toast_compress_datum -
@@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* ----------
*/
static bool
-toastrel_valueid_exists(Relation toastrel, Oid valueid)+toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
{
bool result = false;
ScanKeyData toastkey;
@@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid)
* ----------
*/
static bool
-toastid_valueid_exists(Oid toastrelid, Oid valueid)+toastid_valueid_exists(Oid toastrelid, Oid8 valueid)
{
bool result;
Relation toastrel;
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex 3b7c658a90e6..81154c17376c 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* result is the varlena into which the results should be written.
*/
void
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,+heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
varlena *result)
{
@@ -729,7 +729,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
else
{
/* should never happen */
- elog(ERROR, "found toasted toast chunk for toast value %u in %s",+ elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s",
valueid, RelationGetRelationName(toastrel));
chunksize = 0; /* keep compiler quiet */
chunkdata = NULL;
@@ -741,13 +741,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (curchunk != expectedchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s",+ errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s",
curchunk, expectedchunk, valueid,
RelationGetRelationName(toastrel))));
if (curchunk > endchunk)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s",+ errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s",
curchunk,
startchunk, endchunk, valueid,
RelationGetRelationName(toastrel))));
@@ -756,7 +756,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (chunksize != expected_size)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s",+ errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s",
chunksize, expected_size,
curchunk, totalchunks, valueid,
RelationGetRelationName(toastrel))));
@@ -785,7 +785,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
if (expectedchunk != (endchunk + 1))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg_internal("missing chunk number %d for toast value %u in %s",+ errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.cindex 457d544eec93..dd4b6e8f7f45 100644--- a/src/backend/replication/logical/reorderbuffer.c+++ b/src/backend/replication/logical/reorderbuffer.c@@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
- Oid chunk_id; /* toast_table.chunk_id */+ Oid8 chunk_id; /* toast_table.chunk_id */
int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we
* have seen */
Size num_chunks; /* number of chunks we've already seen */
@@ -5010,7 +5010,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(txn->toast_hash == NULL);
- hash_ctl.keysize = sizeof(Oid);+ hash_ctl.keysize = sizeof(Oid8);
hash_ctl.entrysize = sizeof(ReorderBufferToastEnt);
hash_ctl.hcxt = rb->context;
txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl,
@@ -5034,8 +5034,10 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool isnull;
Pointer chunk;
TupleDesc desc = RelationGetDescr(relation);
- Oid chunk_id;+ Oid8 chunk_id;
int32 chunk_seq;
+ Oid valueid_type;+ Datum valueid_datum;
if (txn->toast_hash == NULL)
ReorderBufferToastInitHash(rb, txn);
@@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(IsToastRelation(relation));
newtup = change->data.tp.newtuple;
- chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull));+ valueid_type = TupleDescAttr(desc, 0)->atttypid;+ valueid_datum = fastgetattr(newtup, 1, desc, &isnull);+ if (valueid_type == OID8OID)+ chunk_id = DatumGetObjectId8(valueid_datum);+ else+ chunk_id = DatumGetObjectId(valueid_datum);
Assert(!isnull);
chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull));
Assert(!isnull);
@@ -5061,11 +5068,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_init(&ent->chunks);
if (chunk_seq != 0)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0",+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0",
chunk_seq, chunk_id);
}
else if (found && chunk_seq != ent->last_chunk_seq + 1)
- elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d",+ elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d",
chunk_seq, chunk_id, ent->last_chunk_seq + 1);
chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull));
@@ -5174,6 +5181,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
+ Oid8 toast_valueid;
if (attr->attisdropped)
continue;
@@ -5194,13 +5202,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
continue;
VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);
+ toast_valueid = toast_pointer.va_valueid;
/*
* Check whether the toast tuple changed, replace if so.
*/
ent = (ReorderBufferToastEnt *)
hash_search(txn->toast_hash,
- &toast_pointer.va_valueid,+ &toast_valueid,
HASH_FIND,
NULL);
if (ent == NULL)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex bbaaec89f8ae..33b00fa4dc5c 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -1564,9 +1564,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 chunksize;
int32 expected_size;
int32 max_chunk_size;
+ Oid8 toast_valueid;
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
+ toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1574,16 +1576,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u has toast chunk with null sequence number",- ta->toast_pointer.va_valueid));+ psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number",+ toast_valueid));
return;
}
if (chunk_seq != *expected_chunk_seq)
{
/* Either the TOAST index is corrupt, or we don't have all chunks. */
report_toast_corruption(ctx, ta,
- psprintf("toast value %u index scan returned chunk %d when expecting chunk %d",- ta->toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d",+ toast_valueid,
chunk_seq, *expected_chunk_seq));
}
*expected_chunk_seq = chunk_seq + 1;
@@ -1594,8 +1596,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (isnull)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has null data",- ta->toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " chunk %d has null data",+ toast_valueid,
chunk_seq));
return;
}
@@ -1614,8 +1616,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has invalid varlena header %0x",- ta->toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x",+ toast_valueid,
chunk_seq, header));
return;
}
@@ -1626,8 +1628,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunk_seq > last_chunk_seq)
{
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d follows last expected chunk %d",- ta->toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d",+ toast_valueid,
chunk_seq, last_chunk_seq));
return;
}
@@ -1637,8 +1639,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
if (chunksize != expected_size)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u chunk %d has size %u, but expected size %u",- ta->toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u",+ toast_valueid,
chunk_seq, chunksize, expected_size));
}
@@ -1669,6 +1671,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
varlena *attr;
char *tp; /* pointer to the tuple data */
uint16 infomask;
+ Oid8 toast_pointer_valueid;
CompactAttribute *thisatt;
varatt_external_oid toast_pointer;
@@ -1777,12 +1780,13 @@ check_tuple_attribute(HeapCheckContext *ctx)
* Must copy attr into toast_pointer for alignment considerations
*/
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+ toast_pointer_valueid = toast_pointer.va_valueid;
/* Toasted attributes too large to be untoasted should never be stored */
if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
- psprintf("toast value %u rawsize %d exceeds limit %d",- toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",+ toast_pointer_valueid,
toast_pointer.va_rawsize,
VARLENA_SIZE_LIMIT));
@@ -1809,16 +1813,16 @@ check_tuple_attribute(HeapCheckContext *ctx)
}
if (!valid)
report_corruption(ctx,
- psprintf("toast value %u has invalid compression method id %d",- toast_pointer.va_valueid, cmid));+ psprintf("toast value " OID8_FORMAT " has invalid compression method id %d",+ toast_pointer_valueid, cmid));
}
/* The tuple header better claim to contain toasted values */
if (!(infomask & HEAP_HASEXTERNAL))
{
report_corruption(ctx,
- psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set",- toast_pointer.va_valueid));+ psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set",+ toast_pointer_valueid));
return true;
}
@@ -1826,8 +1830,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
if (!ctx->rel->rd_rel->reltoastrelid)
{
report_corruption(ctx,
- psprintf("toast value %u is external but relation has no toast relation",- toast_pointer.va_valueid));+ psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation",+ toast_pointer_valueid));
return true;
}
@@ -1873,6 +1877,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
+ Oid8 toast_valueid;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1903,14 +1908,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
+ toast_valueid = ta->toast_pointer.va_valueid;+
if (!found_toasttup)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u not found in toast table",- ta->toast_pointer.va_valueid));+ psprintf("toast value " OID8_FORMAT " not found in toast table",+ toast_valueid));
else if (expected_chunk_seq <= last_chunk_seq)
report_toast_corruption(ctx, ta,
- psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d",- ta->toast_pointer.va_valueid,+ psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d",+ toast_valueid,
last_chunk_seq, expected_chunk_seq));
}
--
2.55.0
v15-0003-Switch-pg_column_toast_chunk_id-return-value-fro.patch (text/plain, 5.7K) download
From 79c4d680b830803acbd02cc5672dba0997fe2ebf Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 13 Aug 2025 19:55:39 +0900
Subject: [PATCH v15 03/11] Switch pg_column_toast_chunk_id() return value from
oid to oid8
This is required for a follow-up patch that will add support for 8-byte
TOAST values, with this function being changed so as it is able to
support the largest TOAST value type available.
XXX: Bump catalog version.
---
src/include/catalog/pg_proc.dat | 2 +-
src/backend/utils/adt/varlena.c | 2 +-
src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +-
src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +-
src/test/modules/injection_points/specs/repack_toast.spec | 2 +-
src/test/regress/expected/misc_functions.out | 2 +-
src/test/regress/sql/misc_functions.sql | 2 +-
doc/src/sgml/func/func-admin.sgml | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex c53ce68c7173..267eaee29806 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -7907,7 +7907,7 @@
proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '6316', descr => 'chunk ID of on-disk TOASTed value',
proname => 'pg_column_toast_chunk_id', provolatile => 's',
- prorettype => 'oid', proargtypes => 'any',+ prorettype => 'oid8', proargtypes => 'any',
prosrc => 'pg_column_toast_chunk_id' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.cindex e5632d930682..1e264904a6e1 100644--- a/src/backend/utils/adt/varlena.c+++ b/src/backend/utils/adt/varlena.c@@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
- PG_RETURN_OID(toast_pointer.va_valueid);+ PG_RETURN_OID8(toast_pointer.va_valueid);
}
/*
diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.outindex cb14ddcee34b..21859fc0dce2 100644--- a/src/test/isolation/expected/cluster-toast-value-reuse.out+++ b/src/test/isolation/expected/cluster-toast-value-reuse.out@@ -21,7 +21,7 @@ step s2_verify_chunk_ids:
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
chunk_ids_preserved
-------------------
diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.specindex 9a2d10600b39..96cf899db296 100644--- a/src/test/isolation/specs/cluster-toast-value-reuse.spec+++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec@@ -60,7 +60,7 @@ step s2_verify_chunk_ids {
SELECT o.id AS chunk_ids_preserved
FROM cluster_chunk_id o
JOIN cluster_toast_value c ON o.id = c.id
- WHERE o.chunk_id != pg_column_toast_chunk_id(c.value);+ WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value);
}
# Run UPDATE with its transaction still open, then store the chunk IDs.
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.specindex fa76bcf28ef9..cc8f034d0167 100644--- a/src/test/modules/injection_points/specs/repack_toast.spec+++ b/src/test/modules/injection_points/specs/repack_toast.spec@@ -59,7 +59,7 @@ setup
CREATE TABLE relfilenodes(node oid);
- CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid);+ CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8);
CREATE TABLE data_s2 (LIKE data_s1);
}
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.outindex 2990e0c4f286..35d6ee2ce9eb 100644--- a/src/test/regress/expected/misc_functions.out+++ b/src/test/regress/expected/misc_functions.out@@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
?column? | ?column?
----------+----------
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sqlindex 950d9ab1a4a1..94b91964c046 100644--- a/src/test/regress/sql/misc_functions.sql+++ b/src/test/regress/sql/misc_functions.sql@@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c
WHERE c.relname = 'test_chunk_id'
\gset
SELECT pg_column_toast_chunk_id(a) IS NULL,
- pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel)+ pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel)
FROM test_chunk_id;
DROP TABLE test_chunk_id;
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgmlindex 54eeb42e5bce..70d762e73328 100644--- a/doc/src/sgml/func/func-admin.sgml+++ b/doc/src/sgml/func/func-admin.sgml@@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
<primary>pg_column_toast_chunk_id</primary>
</indexterm>
<function>pg_column_toast_chunk_id</function> ( <type>"any"</type> )
- <returnvalue>oid</returnvalue>+ <returnvalue>oid8</returnvalue>
</para>
<para>
Shows the <structfield>chunk_id</structfield> of an on-disk
--
2.55.0
v15-0004-Add-support-for-TOAST-chunk_id-type-in-binary-up.patch (text/plain, 9.3K) download
From d0873e44ebadd89ed33d76e0d8817d851ceda046 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 10:57:59 +0900
Subject: [PATCH v15 04/11] Add support for TOAST chunk_id type in binary
upgrades
This commit adds a new function, which would set the type of a chunk_id
attribute for a TOAST table across upgrades. This piece currently works
only with chunk_id = OIDOID, but it is required in a follow-up patch
where support for chunk_id = OID8OID is supported on top of the existing
one.
---
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 ++++
src/backend/catalog/heap.c | 1 +
src/backend/catalog/toasting.c | 20 ++++++++++++++++++-
src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++++
src/bin/pg_dump/pg_dump.c | 10 +++++++++-
.../expected/spgist_name_ops.out | 6 ++++--
7 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.hindex 7bf7ae443859..117f3e3f7746 100644--- a/src/include/catalog/binary_upgrade.h+++ b/src/include/catalog/binary_upgrade.h@@ -29,6 +29,7 @@ extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_index_pg_class_relfilenumber;
extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber;
+extern PGDLLIMPORT Oid binary_upgrade_next_toast_chunk_id_typoid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 267eaee29806..02602d96a2be 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -11992,6 +11992,10 @@
proname => 'binary_upgrade_set_next_toast_pg_class_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_toast_pg_class_oid' },
+{ oid => '8219', descr => 'for use by pg_upgrade',+ proname => 'binary_upgrade_set_next_toast_chunk_id_typoid', provolatile => 'v',+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',+ prosrc => 'binary_upgrade_set_next_toast_chunk_id_typoid' },
{ oid => '3589', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_set_next_pg_enum_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.cindex a107193d7ab3..1c188b7a0ff3 100644--- a/src/backend/catalog/heap.c+++ b/src/backend/catalog/heap.c@@ -80,6 +80,7 @@
/* Potentially set by pg_upgrade_support functions */
Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid;
Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid;
+Oid binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
RelFileNumber binary_upgrade_next_heap_pg_class_relfilenumber = InvalidRelFileNumber;
RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumber = InvalidRelFileNumber;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.cindex 4aa52a4bd253..bc0a72268335 100644--- a/src/backend/catalog/toasting.c+++ b/src/backend/catalog/toasting.c@@ -146,6 +146,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
int16 coloptions[2];
ObjectAddress baseobject,
toastobject;
+ Oid toast_chunkid_typid = OIDOID;
/*
* Is it already toasted?
@@ -184,6 +185,23 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!OidIsValid(binary_upgrade_next_toast_pg_class_oid))
return false;
++ /*+ * The attribute type for chunk_id should have been set when requesting+ * a TOAST table creation.+ */+ if (!OidIsValid(binary_upgrade_next_toast_chunk_id_typoid))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("toast chunk_id type not set while in binary upgrade mode")));+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("cannot support toast chunk_id type %u in binary upgrade mode",+ binary_upgrade_next_toast_chunk_id_typoid)));++ toast_chunkid_typid = binary_upgrade_next_toast_chunk_id_typoid;+ binary_upgrade_next_toast_chunk_id_typoid = InvalidOid;
}
/*
@@ -205,7 +223,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
"chunk_id",
- OIDOID,+ toast_chunkid_typid,
-1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2,
"chunk_seq",
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.cindex b505a6b4feeb..06b789d71ded 100644--- a/src/backend/utils/adt/pg_upgrade_support.c+++ b/src/backend/utils/adt/pg_upgrade_support.c@@ -149,6 +149,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum+binary_upgrade_set_next_toast_chunk_id_typoid(PG_FUNCTION_ARGS)+{+ Oid typoid = PG_GETARG_OID(0);++ CHECK_IS_BINARY_UPGRADE;+ binary_upgrade_next_toast_chunk_id_typoid = typoid;++ PG_RETURN_VOID();+}+
Datum
binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.cindex 7a12e03ca028..bb17a2b52400 100644--- a/src/bin/pg_dump/pg_dump.c+++ b/src/bin/pg_dump/pg_dump.c@@ -104,6 +104,7 @@ typedef struct
RelFileNumber relfilenumber; /* object filenode */
Oid toast_oid; /* toast table OID */
RelFileNumber toast_relfilenumber; /* toast table filenode */
+ Oid toast_chunk_id_typoid; /* type of chunk_id attribute */
Oid toast_index_oid; /* toast table index OID */
RelFileNumber toast_index_relfilenumber; /* toast table index filenode */
} BinaryUpgradeClassOidItem;
@@ -5869,7 +5870,10 @@ collectBinaryUpgradeClassOids(Archive *fout)
const char *query;
query = "SELECT c.oid, c.relkind, c.relfilenode, c.reltoastrelid, "
- "ct.relfilenode, i.indexrelid, cti.relfilenode "+ "ct.relfilenode, i.indexrelid, cti.relfilenode, "+ "(SELECT a.atttypid FROM pg_attribute AS a "+ " WHERE a.attrelid = c.reltoastrelid AND attname = 'chunk_id'::text) "+ " AS toastchunktypid "
"FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_index i "
"ON (c.reltoastrelid = i.indrelid AND i.indisvalid) "
"LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) "
@@ -5891,6 +5895,7 @@ collectBinaryUpgradeClassOids(Archive *fout)
binaryUpgradeClassOids[i].toast_relfilenumber = atooid(PQgetvalue(res, i, 4));
binaryUpgradeClassOids[i].toast_index_oid = atooid(PQgetvalue(res, i, 5));
binaryUpgradeClassOids[i].toast_index_relfilenumber = atooid(PQgetvalue(res, i, 6));
+ binaryUpgradeClassOids[i].toast_chunk_id_typoid = atooid(PQgetvalue(res, i, 7));
}
PQclear(res);
@@ -5955,6 +5960,9 @@ binary_upgrade_set_pg_class_oids(Archive *fout,
appendPQExpBuffer(upgrade_buffer,
"SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n",
entry->toast_relfilenumber);
+ appendPQExpBuffer(upgrade_buffer,+ "SELECT pg_catalog.binary_upgrade_set_next_toast_chunk_id_typoid('%u'::pg_catalog.oid);\n",+ entry->toast_chunk_id_typoid);
/* every toast table has an index */
appendPQExpBuffer(upgrade_buffer,
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.outindex 1ee65ede2430..35e59d0cd83c 100644--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out@@ -61,9 +61,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -110,9 +111,10 @@ select * from t
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
+ binary_upgrade_set_next_toast_chunk_id_typoid | | binary_upgrade_set_next_toast_chunk_id_typoid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.55.0
v15-0005-Add-relation-option-toast_value_type.patch (text/plain, 6.4K) download
From 296ca1549a4aff6ecc4c2bac4602335f1a3566b2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 14 Aug 2025 12:54:58 +0900
Subject: [PATCH v15 05/11] Add relation option toast_value_type
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is created
initially. This parameter has no effect if a TOAST table exists, even
after it is modified later on, even on rewrites.
This can be set only to "oid" currently, and will be expanded with a
second mode later.
Note: perhaps it would make sense to introduce that only when support
for 8-byte OID values are added to TOAST, the split is here to ease
review.
---
src/include/utils/rel.h | 17 +++++++++++++++++
src/backend/access/common/reloptions.c | 21 +++++++++++++++++++++
src/backend/catalog/toasting.c | 6 ++++++
src/bin/psql/tab-complete.in.c | 1 +
doc/src/sgml/ref/create_table.sgml | 18 ++++++++++++++++++
5 files changed, 63 insertions(+)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex 41ab4586c6b5..f9e26ce6654a 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -341,11 +341,20 @@ typedef enum StdRdOptIndexCleanup
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
} StdRdOptIndexCleanup;
+/* StdRdOptions->toast_value_type values */+typedef enum StdRdOptToastValueType+{+ STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,+} StdRdOptToastValueType;+
typedef struct StdRdOptions
{
int32 vl_len_; /* varlena header (do not touch directly!) */
int fillfactor; /* page fill factor in percent (0..100) */
int toast_tuple_target; /* target for tuple toasting */
+ StdRdOptToastValueType toast_value_type; /* type assigned to chunk_id+ * at toast table creation */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
int parallel_workers; /* max number of parallel workers */
@@ -370,6 +379,14 @@ typedef struct StdRdOptions
((relation)->rd_options ? \
((StdRdOptions *) (relation)->rd_options)->toast_tuple_target : (defaulttarg))
+/*+ * RelationGetToastValueType+ * Returns the relation's toast_value_type. Note multiple eval of argument!+ */+#define RelationGetToastValueType(relation, defaulttarg) \+ ((relation)->rd_options ? \+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)+
/*
* RelationGetFillFactor
* Returns the relation's fillfactor. Note multiple eval of argument!
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.cindex c3f6f45f4609..82491a31a9a4 100644--- a/src/backend/access/common/reloptions.c+++ b/src/backend/access/common/reloptions.c@@ -548,6 +548,14 @@ static relopt_enum_elt_def viewCheckOptValues[] =
{(const char *) NULL} /* list terminator */
};
+/* values from StdRdOptToastValueType */+static relopt_enum_elt_def StdRdOptToastValueTypes[] =+{+ /* no value for INVALID */+ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},+ {(const char *) NULL} /* list terminator */+};+
static relopt_enum enumRelOpts[] =
{
{
@@ -561,6 +569,17 @@ static relopt_enum enumRelOpts[] =
STDRD_OPTION_VACUUM_INDEX_CLEANUP_NOT_SET,
gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")
},
+ {+ {+ "toast_value_type",+ "Controls the attribute type of chunk_id at toast table creation",+ RELOPT_KIND_HEAP,+ ShareUpdateExclusiveLock+ },+ StdRdOptToastValueTypes,+ STDRD_OPTION_TOAST_VALUE_TYPE_OID,+ gettext_noop("Valid values are \"oid\".")+ },
{
{
"buffering",
@@ -2078,6 +2097,8 @@ static const relopt_parse_elt stdRdOptionsTab[] = {
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_analyze_min_duration)},
{"toast_tuple_target", RELOPT_TYPE_INT,
offsetof(StdRdOptions, toast_tuple_target)},
+ {"toast_value_type", RELOPT_TYPE_ENUM,+ offsetof(StdRdOptions, toast_value_type)},
{"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL,
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)},
{"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL,
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.cindex bc0a72268335..28ddbd10d357 100644--- a/src/backend/catalog/toasting.c+++ b/src/backend/catalog/toasting.c@@ -159,9 +159,15 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
*/
if (!IsBinaryUpgrade)
{
+ StdRdOptToastValueType value_type;+
/* Normal mode, normal check */
if (!needs_toast_table(rel))
return false;
++ value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);+ if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)+ toast_chunkid_typid = OIDOID;
}
else
{
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.cindex 8e1a1d69740c..84a129efff54 100644--- a/src/bin/psql/tab-complete.in.c+++ b/src/bin/psql/tab-complete.in.c@@ -1441,6 +1441,7 @@ static const char *const table_storage_parameters[] = {
"toast.vacuum_max_eager_freeze_failure_rate",
"toast.vacuum_truncate",
"toast_tuple_target",
+ "toast_value_type",
"user_catalog_table",
"vacuum_index_cleanup",
"vacuum_max_eager_freeze_failure_rate",
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgmlindex e342585c7f08..3da754045c04 100644--- a/doc/src/sgml/ref/create_table.sgml+++ b/doc/src/sgml/ref/create_table.sgml@@ -1653,6 +1653,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry id="reloption-toast-value-type" xreflabel="toast_value_type">+ <term><literal>toast_value_type</literal> (<type>enum</type>)+ <indexterm>+ <primary><varname>toast_value_type</varname> storage parameter</primary>+ </indexterm>+ </term>+ <listitem>+ <para>+ The toast_value_type specifies the attribute type of+ <literal>chunk_id</literal> used when initially creating a toast+ relation for this table.+ By default this parameter is <literal>oid</literal>, to assign+ <type>oid</type> as attribute type to <literal>chunk_id</literal>.+ This parameter cannot be set for TOAST tables.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry id="reloption-parallel-workers" xreflabel="parallel_workers">
<term><literal>parallel_workers</literal> (<type>integer</type>)
<indexterm>
--
2.55.0
v15-0006-Add-support-for-oid8-TOAST-values.patch (text/plain, 17.4K) download
From a8f000c92bae94d41559f933c27c56f134b30356 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:45:13 +0900
Subject: [PATCH v15 06/11] Add support for oid8 TOAST values
This commit adds the possibility to define TOAST tables with oid8 as
value ID, based on the reloption toast_value_type. All the external
TOAST pointers still rely on varatt_external and a single vartag.
The values inserted in the oid8 TOAST tables are fed from the OID8
value generator, casted to OID (for now).
This will be changed in an upcoming patch that adds more vartag_external
types and its associated structures, with the code being able to use a
different external TOAST pointer depending on the attribute type of
chunk_id in TOAST relations.
All the changes done here are mechanical, with all the TOAST code able
to do chunk ID lookups based on the two types now supported.
XXX: Catalog version bump required.
---
src/include/catalog/pg_opclass.dat | 3 +-
src/include/utils/rel.h | 1 +
src/backend/access/common/reloptions.c | 3 +-
src/backend/access/common/toast_internals.c | 94 +++++++++++++++------
src/backend/access/heap/heaptoast.c | 19 ++++-
src/backend/catalog/toasting.c | 24 +++++-
doc/src/sgml/ref/create_table.sgml | 2 +
doc/src/sgml/storage.sgml | 7 +-
contrib/amcheck/verify_heapam.c | 22 +++--
9 files changed, 133 insertions(+), 42 deletions(-)
diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.datindex df170b80840b..b84c2bb7a8c3 100644--- a/src/include/catalog/pg_opclass.dat+++ b/src/include/catalog/pg_opclass.dat@@ -179,7 +179,8 @@
opcintype => 'xid8' },
{ opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops',
opcintype => 'oid8' },
-{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',+{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID',+ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops',
opcintype => 'oid8' },
{ opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops',
opcintype => 'cid' },
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex f9e26ce6654a..b7da03446a94 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType
{
STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
+ STDRD_OPTION_TOAST_VALUE_TYPE_OID8,
} StdRdOptToastValueType;
typedef struct StdRdOptions
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.cindex 82491a31a9a4..ea9a04179090 100644--- a/src/backend/access/common/reloptions.c+++ b/src/backend/access/common/reloptions.c@@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] =
{
/* no value for INVALID */
{"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID},
+ {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8},
{(const char *) NULL} /* list terminator */
};
@@ -578,7 +579,7 @@ static relopt_enum enumRelOpts[] =
},
StdRdOptToastValueTypes,
STDRD_OPTION_TOAST_VALUE_TYPE_OID,
- gettext_noop("Valid values are \"oid\".")+ gettext_noop("Valid values are \"oid\" and \"oid8\".")
},
{
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex ca5ed8dfabca..2c180491c2ff 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -25,6 +25,7 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,8 +132,10 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
+ Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
Assert(!VARATT_IS_EXTERNAL(dval));
+ Assert(OidIsValid(toast_typid));
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -200,24 +203,32 @@ toast_save_datum(Relation rel, Datum value,
toast_pointer.va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose an OID to use as the value ID for this toast value.+ * Choose a new value to use as the value ID for this toast value, be it+ * for OID or OID8 TOAST relations.
*
- * Normally we just choose an unused OID within the toast table. But+ * Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
- * toast table OID, we want to preserve toast value OIDs too. So, if+ * toast table OID, we want to preserve toast value IDs too. So, if
* rd_toastoid is set and we had a prior external value from that same
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value OIDs.+ * conflict with either new or existing toast value IDs. If the TOAST+ * table uses 8-byte value IDs, we should not really care much about+ * that.
*/
if (!OidIsValid(rel->rd_toastoid))
{
/* normal case: just choose an unused OID */
- toast_pointer.va_valueid =- GetNewOidWithIndex(toastrel,- RelationGetRelid(toastidxs[validIndex]),- (AttrNumber) 1);+ if (toast_typid == OIDOID)+ toast_pointer.va_valueid =+ GetNewOidWithIndex(toastrel,+ RelationGetRelid(toastidxs[validIndex]),+ (AttrNumber) 1);+ else if (toast_typid == OID8OID)+ toast_pointer.va_valueid = GetNewObjectId8();+ else+ Assert(false);
}
else
{
@@ -263,17 +274,22 @@ toast_save_datum(Relation rel, Datum value,
if (toast_pointer.va_valueid == InvalidOid)
{
/*
- * new value; must choose an OID that doesn't conflict in either- * old or new toast table+ * new value; must choose a value that doesn't conflict in either+ * old or new toast table.
*/
- do+ if (toast_typid == OIDOID)
{
- toast_pointer.va_valueid =- GetNewOidWithIndex(toastrel,- RelationGetRelid(toastidxs[validIndex]),- (AttrNumber) 1);- } while (toastid_valueid_exists(rel->rd_toastoid,- toast_pointer.va_valueid));+ do+ {+ toast_pointer.va_valueid =+ GetNewOidWithIndex(toastrel,+ RelationGetRelid(toastidxs[validIndex]),+ (AttrNumber) 1);+ } while (toastid_valueid_exists(rel->rd_toastoid,+ toast_pointer.va_valueid));+ }+ else if (toast_typid == OID8OID)+ toast_pointer.va_valueid = GetNewObjectId8();
}
}
@@ -303,7 +319,10 @@ toast_save_datum(Relation rel, Datum value,
/*
* Build a tuple and store it
*/
- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);+ if (toast_typid == OIDOID)+ t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);+ else if (toast_typid == OID8OID)+ t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -384,6 +403,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
+ Oid toast_typid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
@@ -395,6 +415,8 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
* Open the toast relation and its indexes
*/
toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/* Fetch valid relation used for process */
validIndex = toast_open_indexes(toastrel,
@@ -405,10 +427,18 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(toast_pointer.va_valueid));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(toast_pointer.va_valueid));+ else if (toast_typid == OID8OID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OID8EQ,+ ObjectId8GetDatum(toast_pointer.va_valueid));+ else+ Assert(false);
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -455,6 +485,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
int num_indexes;
int validIndex;
Relation *toastidxs;
+ Oid toast_typid;
/* Fetch a valid index relation */
validIndex = toast_open_indexes(toastrel,
@@ -462,13 +493,24 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);+
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(valueid));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(valueid));+ else if (toast_typid == OID8OID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OID8EQ,+ ObjectId8GetDatum(valueid));+ else+ Assert(false);
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex 81154c17376c..a93e98788da7 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
int num_indexes;
int validIndex;
int32 max_chunk_size;
+ Oid toast_typid;
/* Look for the valid index of toast relation */
validIndex = toast_open_indexes(toastrel,
@@ -647,6 +648,8 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
&toastidxs,
&num_indexes);
+ toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
@@ -655,10 +658,18 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- ScanKeyInit(&toastkey[0],- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(valueid));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey[0],+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(valueid));+ else if (toast_typid == OID8OID)+ ScanKeyInit(&toastkey[0],+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OID8EQ,+ ObjectId8GetDatum(valueid));+ else+ Assert(false);
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.cindex 28ddbd10d357..66d483bb1be3 100644--- a/src/backend/catalog/toasting.c+++ b/src/backend/catalog/toasting.c@@ -32,6 +32,7 @@
#include "nodes/makefuncs.h"
#include "utils/fmgroids.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
static void CheckAndCreateToastTable(Oid relOid, Datum reloptions,
@@ -168,6 +169,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)
toast_chunkid_typid = OIDOID;
+ else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)+ toast_chunkid_typid = OID8OID;
}
else
{
@@ -200,7 +203,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("toast chunk_id type not set while in binary upgrade mode")));
- if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID)+ if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID &&+ binary_upgrade_next_toast_chunk_id_typoid != OID8OID)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot support toast chunk_id type %u in binary upgrade mode",
@@ -225,6 +229,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
snprintf(toast_idxname, sizeof(toast_idxname),
"pg_toast_%u_index", relOid);
+ /*+ * Special case here. If OIDOldToast is defined, we need to rely on the+ * existing table for the job because we do not want to create an+ * inconsistent relation that would conflict with the parent and break+ * the world.+ */+ if (OidIsValid(OIDOldToast))+ {+ toast_chunkid_typid = get_atttype(OIDOldToast, 1);+ if (!OidIsValid(toast_chunkid_typid))+ elog(ERROR, "cache lookup failed for relation %u", OIDOldToast);+ }+
/* this is pretty painful... need a tuple descriptor */
tupdesc = CreateTemplateTupleDesc(3);
TupleDescInitEntry(tupdesc, (AttrNumber) 1,
@@ -343,7 +360,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
collationIds[0] = InvalidOid;
collationIds[1] = InvalidOid;
- opclassIds[0] = OID_BTREE_OPS_OID;+ if (toast_chunkid_typid == OIDOID)+ opclassIds[0] = OID_BTREE_OPS_OID;+ else if (toast_chunkid_typid == OID8OID)+ opclassIds[0] = OID8_BTREE_OPS_OID;
opclassIds[1] = INT4_BTREE_OPS_OID;
coloptions[0] = 0;
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgmlindex 3da754045c04..3ed8f3581177 100644--- a/doc/src/sgml/ref/create_table.sgml+++ b/doc/src/sgml/ref/create_table.sgml@@ -1666,6 +1666,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
relation for this table.
By default this parameter is <literal>oid</literal>, to assign
<type>oid</type> as attribute type to <literal>chunk_id</literal>.
+ This parameter can be set to <type>oid8</type> to use <type>oid8</type>+ as attribute type for <literal>chunk_id</literal>.
This parameter cannot be set for TOAST tables.
</para>
</listitem>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 90aae3defcbc..21bd67c11ea0 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
-<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID-identifying the particular <acronym>TOAST</acronym>ed value),+<acronym>TOAST</acronym> table has the columns+<structfield>chunk_id</structfield> (an OID or an OID8 identifying+the particular <acronym>TOAST</acronym>ed value),
<structfield>chunk_seq</structfield> (a sequence number for the chunk within its value),
and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index
on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast
retrieval of the values. A pointer datum representing an out-of-line on-disk
<acronym>TOAST</acronym>ed value therefore needs to store the OID of the
-<acronym>TOAST</acronym> table in which to look and the OID of the specific value+<acronym>TOAST</acronym> table in which to look and the specific value
(its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 33b00fa4dc5c..42f409548e1d 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -1876,8 +1876,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
uint32 extsize;
int32 expected_chunk_seq = 0;
int32 last_chunk_seq;
- int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;+ int32 max_chunk_size;
Oid8 toast_valueid;
+ Oid toast_typid;++ toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);
last_chunk_seq = (extsize - 1) / max_chunk_size;
@@ -1885,10 +1889,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(ta->toast_pointer.va_valueid));+ if (toast_typid == OIDOID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(ta->toast_pointer.va_valueid));+ else if (toast_typid == OID8OID)+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OID8EQ,+ ObjectId8GetDatum(ta->toast_pointer.va_valueid));+ else+ Assert(false);
/*
* Check if any chunks for this toasted object exist in the toast table,
--
2.55.0
v15-0007-Add-tests-for-TOAST-relations-with-oid8-as-value.patch (text/plain, 21.1K) download
From a963f46f2071206a20f3b0b229ec9bd4b8d6de4c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 08:22:56 +0900
Subject: [PATCH v15 07/11] Add tests for TOAST relations with oid8 as value
type
This adds coverage for relations created with default_toast_type =
'oid8', for external TOAST pointers both compressed and uncompressed.
---
src/test/regress/expected/strings.out | 257 ++++++++++++++++++++++----
src/test/regress/sql/strings.sql | 156 ++++++++++++----
2 files changed, 338 insertions(+), 75 deletions(-)
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.outindex 313c50394650..080cb039ce43 100644--- a/src/test/regress/expected/strings.out+++ b/src/test/regress/expected/strings.out@@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
(1 row)
--
--- test substr with toasted text values+-- Test substr with toasted text values, for all types of TOAST relations+-- supported.
--
-CREATE TABLE toasttest(f1 text);-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid8 values(repeat('1234567890',10000));+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+alter table toasttest_oid8 alter column f1 set storage external;+insert into toasttest_oid8 values(repeat('1234567890',10000));+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+ substr +--------+ 123+ 123+ 123+ 123+(4 rows)++SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+ERROR: negative substring length not allowed+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-TRUNCATE TABLE toasttest;-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+SELECT substr(f1, 99995, 10) from toasttest_oid8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++-- TRUNCATE cases for TOAST relations with OID values.+TRUNCATE TABLE toasttest_oid;+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
f
(1 row)
-TRUNCATE TABLE toasttest;-ALTER TABLE toasttest set (toast_tuple_target = 4080);-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+TRUNCATE TABLE toasttest_oid;+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';+ FROM pg_class where relname = 'toasttest_oid';
is_empty
----------
t
(1 row)
-DROP TABLE toasttest;+DROP TABLE toasttest_oid;+-- TRUNCATE cases for TOAST relation with int8 values.+TRUNCATE TABLE toasttest_oid8;+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+-- expect >0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_oid8';+ is_empty +----------+ f+(1 row)++TRUNCATE TABLE toasttest_oid8;+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+-- expect 0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_oid8';+ is_empty +----------+ t+(1 row)++DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values+-- test substr with toasted bytea values, for all types of TOAST relations+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_oid8 alter column f1 set storage external;+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+ substr +--------+ 123+ 123+ 123+ 123+(4 rows)++SELECT substr(f1, -1, 5) from toasttest_oid8;
substr
--------
123
@@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest;
(4 rows)
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+ERROR: negative substring length not allowed+SELECT substr(f1, 5, -1) from toasttest_oid8;
ERROR: negative substring length not allowed
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995) from toasttest_oid8;
substr
--------
567890
@@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;
substr
--------
567890
@@ -2105,7 +2198,97 @@ SELECT substr(f1, 99995, 10) from toasttest;
567890
(4 rows)
-DROP TABLE toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++-- A relation rewrite leaves the TOAST value attributes unchanged.+VACUUM FULL toasttest_oid;+VACUUM FULL toasttest_oid8;+SELECT c1.relname, a.atttypid::regtype+ FROM pg_attribute AS a,+ pg_class AS c1,+ pg_class AS c2+ WHERE+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND+ c1.reltoastrelid = c2.oid AND+ a.attrelid = c2.oid AND+ a.attname = 'chunk_id'+ ORDER BY c1.relname COLLATE "C";+ relname | atttypid +----------------+----------+ toasttest_oid | oid+ toasttest_oid8 | oid8+(2 rows)++-- Check that data slices are still accessible.+SELECT substr(f1, 99995) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995) from toasttest_oid8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995, 10) from toasttest_oid;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++SELECT substr(f1, 99995, 10) from toasttest_oid8;+ substr +--------+ 567890+ 567890+ 567890+ 567890+(4 rows)++-- ALTER TABLE after TOAST table creation does not affect relation rewrite.+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);+VACUUM FULL toasttest_oid;+VACUUM FULL toasttest_oid8;+SELECT c1.relname, a.atttypid::regtype+ FROM pg_attribute AS a,+ pg_class AS c1,+ pg_class AS c2+ WHERE+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND+ c1.reltoastrelid = c2.oid AND+ a.attrelid = c2.oid AND+ a.attname = 'chunk_id'+ ORDER BY c1.relname COLLATE "C";+ relname | atttypid +----------------+----------+ toasttest_oid | oid+ toasttest_oid8 | oid8+(2 rows)++ALTER TABLE toasttest_oid RESET (toast_value_type);+ALTER TABLE toasttest_oid8 RESET (toast_value_type);+-- Reset column storage to its default+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sqlindex 38946e8954df..3e7459d29ff1 100644--- a/src/test/regress/sql/strings.sql+++ b/src/test/regress/sql/strings.sql@@ -563,89 +563,169 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char";
SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar";
--
--- test substr with toasted text values+-- Test substr with toasted text values, for all types of TOAST relations+-- supported.
--
-CREATE TABLE toasttest(f1 text);+CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid');+CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8');-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid8 values(repeat('1234567890',10000));+insert into toasttest_oid8 values(repeat('1234567890',10000));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(repeat('1234567890',10000));-insert into toasttest values(repeat('1234567890',10000));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(repeat('1234567890',10000));+insert into toasttest_oid values(repeat('1234567890',10000));+alter table toasttest_oid8 alter column f1 set storage external;+insert into toasttest_oid8 values(repeat('1234567890',10000));+insert into toasttest_oid8 values(repeat('1234567890',10000));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;+SELECT substr(f1, 99995, 10) from toasttest_oid8;-TRUNCATE TABLE toasttest;-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+-- TRUNCATE cases for TOAST relations with OID values.+TRUNCATE TABLE toasttest_oid;+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect >0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';--TRUNCATE TABLE toasttest;-ALTER TABLE toasttest set (toast_tuple_target = 4080);-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));-INSERT INTO toasttest values (repeat('1234567890',300));+ FROM pg_class where relname = 'toasttest_oid';+TRUNCATE TABLE toasttest_oid;+ALTER TABLE toasttest_oid set (toast_tuple_target = 4080);+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));+INSERT INTO toasttest_oid values (repeat('1234567890',300));
-- expect 0 blocks
SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty
- FROM pg_class where relname = 'toasttest';+ FROM pg_class where relname = 'toasttest_oid';+DROP TABLE toasttest_oid;-DROP TABLE toasttest;+-- TRUNCATE cases for TOAST relation with int8 values.+TRUNCATE TABLE toasttest_oid8;+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+-- expect >0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_oid8';+TRUNCATE TABLE toasttest_oid8;+ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080);+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+INSERT INTO toasttest_oid8 values (repeat('1234567890',300));+-- expect 0 blocks+SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty+ FROM pg_class where relname = 'toasttest_oid8';+DROP TABLE toasttest_oid8;
--
--- test substr with toasted bytea values+-- test substr with toasted bytea values, for all types of TOAST relations+-- supported. Do not drop these two relations, for pg_upgrade.
--
-CREATE TABLE toasttest(f1 bytea);+CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid');+CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8');-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
--
-- Ensure that some values are uncompressed, to test the faster substring
-- operation used in that case
--
-alter table toasttest alter column f1 set storage external;-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));-insert into toasttest values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_oid alter column f1 set storage external;+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape'));+alter table toasttest_oid8 alter column f1 set storage external;+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));+insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape'));
-- If the starting position is zero or less, then return from the start of the string
-- adjusting the length to be consistent with the "negative start" per SQL.
-SELECT substr(f1, -1, 5) from toasttest;+SELECT substr(f1, -1, 5) from toasttest_oid;+SELECT substr(f1, -1, 5) from toasttest_oid8;
-- If the length is less than zero, an ERROR is thrown.
-SELECT substr(f1, 5, -1) from toasttest;+SELECT substr(f1, 5, -1) from toasttest_oid;+SELECT substr(f1, 5, -1) from toasttest_oid8;
-- If no third argument (length) is provided, the length to the end of the
-- string is assumed.
-SELECT substr(f1, 99995) from toasttest;+SELECT substr(f1, 99995) from toasttest_oid;+SELECT substr(f1, 99995) from toasttest_oid8;
-- If start plus length is > string length, the result is truncated to
-- string length
-SELECT substr(f1, 99995, 10) from toasttest;+SELECT substr(f1, 99995, 10) from toasttest_oid;+SELECT substr(f1, 99995, 10) from toasttest_oid8;-DROP TABLE toasttest;+-- A relation rewrite leaves the TOAST value attributes unchanged.+VACUUM FULL toasttest_oid;+VACUUM FULL toasttest_oid8;+SELECT c1.relname, a.atttypid::regtype+ FROM pg_attribute AS a,+ pg_class AS c1,+ pg_class AS c2+ WHERE+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND+ c1.reltoastrelid = c2.oid AND+ a.attrelid = c2.oid AND+ a.attname = 'chunk_id'+ ORDER BY c1.relname COLLATE "C";+-- Check that data slices are still accessible.+SELECT substr(f1, 99995) from toasttest_oid;+SELECT substr(f1, 99995) from toasttest_oid8;+SELECT substr(f1, 99995, 10) from toasttest_oid;+SELECT substr(f1, 99995, 10) from toasttest_oid8;++-- ALTER TABLE after TOAST table creation does not affect relation rewrite.+ALTER TABLE toasttest_oid SET (toast_value_type = oid8);+ALTER TABLE toasttest_oid8 SET (toast_value_type = oid);+VACUUM FULL toasttest_oid;+VACUUM FULL toasttest_oid8;+SELECT c1.relname, a.atttypid::regtype+ FROM pg_attribute AS a,+ pg_class AS c1,+ pg_class AS c2+ WHERE+ c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND+ c1.reltoastrelid = c2.oid AND+ a.attrelid = c2.oid AND+ a.attname = 'chunk_id'+ ORDER BY c1.relname COLLATE "C";+ALTER TABLE toasttest_oid RESET (toast_value_type);+ALTER TABLE toasttest_oid8 RESET (toast_value_type);++-- Reset column storage to its default+ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;+ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
-- test internally compressing datums
--
2.55.0
v15-0008-More-test-coverage-for-TOAST-relations-with-oid8.patch (text/plain, 13.4K) download
From 043fcf768f069908bce9a26ac7b398edcf7eaa2a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:46:39 +0900
Subject: [PATCH v15 08/11] More test coverage for TOAST relations with oid8 as
chunk_id
This covers more area of the code in terms of tests, where oid8 vartags
are added:
- amcheck- test_decoding- reloptions, with various incorrect patterns for toast_value_type.---
src/test/regress/expected/reloptions.out | 35 +++++++++++++++++++++
src/test/regress/sql/reloptions.sql | 20 ++++++++++++
contrib/amcheck/expected/check_heap.out | 40 ++++++++++++++++++++++++
contrib/amcheck/sql/check_heap.sql | 27 ++++++++++++++++
contrib/test_decoding/expected/toast.out | 39 +++++++++++++++++++++++
contrib/test_decoding/sql/toast.sql | 20 ++++++++++++
6 files changed, 181 insertions(+)
diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.outindex e3a974f26112..6a50713fb58e 100644--- a/src/test/regress/expected/reloptions.out+++ b/src/test/regress/expected/reloptions.out@@ -194,6 +194,41 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid;
-- Fail on non-existent options in toast namespace
CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42);
ERROR: unrecognized parameter "not_existing_option"
+-- Test toast_value_type.+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid8');+ERROR: unrecognized parameter "toast_value_type"+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');+ERROR: invalid value for enum option "toast_value_type": int8+DETAIL: Valid values are "oid" and "oid8".+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid8');+SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;+ reloptions +-------------------------+ {toast_value_type=oid8}+(1 row)++-- The option is only consulted when the TOAST relation is created, so+-- changing it afterwards leaves chunk_id alone.+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid');+SELECT a.atttypid::regtype AS chunk_id_type+ FROM pg_class AS c, pg_attribute AS a+ WHERE c.oid = 'reloptions_test2'::regclass AND+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';+ chunk_id_type +---------------+ oid8+(1 row)++ALTER TABLE reloptions_test2 RESET (toast_value_type);+DROP TABLE reloptions_test2;+-- Non-supported relkinds.+CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)+ WITH (toast_value_type = 'oid8');+ERROR: unrecognized parameter "toast_value_type"+CREATE TABLE reloptions_test2 (i int) PARTITION BY RANGE (i)+ WITH (toast_value_type = 'oid8');+ERROR: cannot specify storage parameters for a partitioned table+HINT: Specify storage parameters for its leaf partitions instead.
-- Mix TOAST & heap
DROP TABLE reloptions_test;
CREATE TABLE reloptions_test (s VARCHAR) WITH
diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sqlindex 680c8bf86148..53cad02f2a28 100644--- a/src/test/regress/sql/reloptions.sql+++ b/src/test/regress/sql/reloptions.sql@@ -112,6 +112,26 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid;
-- Fail on non-existent options in toast namespace
CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42);
+-- Test toast_value_type.+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid8');+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8');+CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid8');+SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass;+-- The option is only consulted when the TOAST relation is created, so+-- changing it afterwards leaves chunk_id alone.+ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid');+SELECT a.atttypid::regtype AS chunk_id_type+ FROM pg_class AS c, pg_attribute AS a+ WHERE c.oid = 'reloptions_test2'::regclass AND+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id';+ALTER TABLE reloptions_test2 RESET (toast_value_type);+DROP TABLE reloptions_test2;+-- Non-supported relkinds.+CREATE INDEX reloptions_test_idx0 ON reloptions_test (s)+ WITH (toast_value_type = 'oid8');+CREATE TABLE reloptions_test2 (i int) PARTITION BY RANGE (i)+ WITH (toast_value_type = 'oid8');+
-- Mix TOAST & heap
DROP TABLE reloptions_test;
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.outindex 979e5e84e723..8686fba712d0 100644--- a/contrib/amcheck/expected/check_heap.out+++ b/contrib/amcheck/expected/check_heap.out@@ -199,6 +199,44 @@ SELECT * FROM verify_heapam('test_partition',
-------+--------+--------+-----
(0 rows)
+-- Check TOAST relations of both chunk_id: oid and oid8+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');+-- Uncompressed out-of-line values+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;+INSERT INTO test_toast_oid (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);+INSERT INTO test_toast_oid8 (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);+-- Compressed out-of-line values.+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;+INSERT INTO test_toast_oid (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);+INSERT INTO test_toast_oid8 (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);+SELECT c.relname, a.atttypid::regtype AS chunk_id_type+ FROM pg_class AS c, pg_attribute AS a+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'+ ORDER BY c.relname COLLATE "C";+ relname | chunk_id_type +-----------------+---------------+ test_toast_oid | oid+ test_toast_oid8 | oid8+(2 rows)++SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -232,6 +270,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
+DROP TABLE test_toast_oid;+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sqlindex 1745bae634e5..eb46fe710351 100644--- a/contrib/amcheck/sql/check_heap.sql+++ b/contrib/amcheck/sql/check_heap.sql@@ -112,6 +112,31 @@ SELECT * FROM verify_heapam('test_partition',
startblock := NULL,
endblock := NULL);
+-- Check TOAST relations of both chunk_id: oid and oid8+CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid');+CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8');+-- Uncompressed out-of-line values+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL;+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL;+INSERT INTO test_toast_oid (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);+INSERT INTO test_toast_oid8 (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs);+-- Compressed out-of-line values.+ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED;+ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED;+INSERT INTO test_toast_oid (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);+INSERT INTO test_toast_oid8 (a, b)+ (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs);+SELECT c.relname, a.atttypid::regtype AS chunk_id_type+ FROM pg_class AS c, pg_attribute AS a+ WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND+ a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'+ ORDER BY c.relname COLLATE "C";+SELECT * FROM verify_heapam('test_toast_oid', check_toast := true);+SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true);+
-- Check that indexes are rejected
CREATE INDEX test_index ON test_partition (a);
SELECT * FROM verify_heapam('test_index',
@@ -139,6 +164,8 @@ SELECT * FROM verify_heapam('test_foreign_table',
endblock := NULL);
-- cleanup
+DROP TABLE test_toast_oid;+DROP TABLE test_toast_oid8;
DROP TABLE heaptest;
DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.outindex a757e7dc8d54..8bef6cc32af4 100644--- a/contrib/test_decoding/expected/toast.out+++ b/contrib/test_decoding/expected/toast.out@@ -382,6 +382,45 @@ SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot',
COMMIT
(4 rows)
+-- Test decoding of TOAST values with oid8+CREATE TABLE toasted_oid8 (id serial primary key, data text)+ WITH (toast_value_type = 'oid8');+-- uncompressed external toast data+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));+-- compressed external toast data+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));+-- update without changing the toasted column, reported as unchanged+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;+-- update changing the toasted column+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;+DELETE FROM toasted_oid8;+-- Check that the values are reassembled in full.+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,+ length(data) AS len+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');+ shortened | len +------------------------------------------------------------------------------------+--------+ BEGIN | 5+ table public.toasted_oid8: INSERT: id[integer]:1 data[text]:..2345678901234567890' | 200062+ COMMIT | 6+ BEGIN | 5+ table public.toasted_oid8: INSERT: id[integer]:2 data[text]:..2345678901234567890' | 200062+ COMMIT | 6+ BEGIN | 5+ table public.toasted_oid8: UPDATE: old-key: id[integer]:1 ne..nchanged-toast-datum | 116+ COMMIT | 6+ BEGIN | 5+ table public.toasted_oid8: UPDATE: id[integer]:2 data[text]:..bcdefghijabcdefghij' | 200062+ COMMIT | 6+ BEGIN | 5+ table public.toasted_oid8: DELETE: id[integer]:11 | 49+ table public.toasted_oid8: DELETE: id[integer]:2 | 48+ COMMIT | 6+(16 rows)++DROP TABLE toasted_oid8;
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sqlindex d1c560a174d6..8a2c49a5c9a7 100644--- a/contrib/test_decoding/sql/toast.sql+++ b/contrib/test_decoding/sql/toast.sql@@ -324,4 +324,24 @@ INSERT INTO tbl2 VALUES(1);
commit;
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- Test decoding of TOAST values with oid8+CREATE TABLE toasted_oid8 (id serial primary key, data text)+ WITH (toast_value_type = 'oid8');+-- uncompressed external toast data+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL;+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));+-- compressed external toast data+ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED;+INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000));+-- update without changing the toasted column, reported as unchanged+UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1;+-- update changing the toasted column+UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2;+DELETE FROM toasted_oid8;+-- Check that the values are reassembled in full.+SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened,+ length(data) AS len+ FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');+DROP TABLE toasted_oid8;+
SELECT pg_drop_replication_slot('regression_slot');
--
2.55.0
v15-0009-More-test-coverage-for-oid8-with-compression-and.patch (text/plain, 8.4K) download
From 0426e7499bd2b567064a0d657002d83e77e38e6c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 08:43:35 +0900
Subject: [PATCH v15 09/11] More test coverage for oid8 with compression and
out-of-line values
These two gaps have been pointed out by Claude, for the following items:
- pg_column_compression() with oid8 vartags- More interesting: UPDATE with out-of-line TOAST datum that belongs to
a different TOAST table, where toast_tuple_init() may try to compare two
TOAST tuples with a different vartag once the oid8 flavor is introduced.
As of the current state of the code, all vartags are OID-based. A
follow-up patch will complicate that.
---
src/test/regress/expected/compression.out | 41 +++++++++++++++++++++
src/test/regress/expected/strings.out | 43 +++++++++++++++++++++++
src/test/regress/sql/compression.sql | 22 ++++++++++++
src/test/regress/sql/strings.sql | 25 +++++++++++++
4 files changed, 131 insertions(+)
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.outindex 09f198149aa4..e9128193515a 100644--- a/src/test/regress/expected/compression.out+++ b/src/test/regress/expected/compression.out@@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
(1 row)
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;+ uncompressed +--------------+ t+(1 row)++SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;+ uncompressed +--------------+ t+(1 row)++-- out-of-line and compressed.+TRUNCATE toastcomp_oid;+TRUNCATE toastcomp_oid8;+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));+SELECT pg_column_compression(f1) FROM toastcomp_oid;+ pg_column_compression +-----------------------+ pglz+(1 row)++SELECT pg_column_compression(f1) FROM toastcomp_oid8;+ pg_column_compression +-----------------------+ pglz+(1 row)++DROP TABLE toastcomp_oid, toastcomp_oid8;
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.outindex 080cb039ce43..7e3a641259c2 100644--- a/src/test/regress/expected/strings.out+++ b/src/test/regress/expected/strings.out@@ -2289,6 +2289,49 @@ ALTER TABLE toasttest_oid8 RESET (toast_value_type);
-- Reset column storage to its default
ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class+ WHERE oid = 'toastupd_oid'::regclass \gset+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class+ WHERE oid = 'toastupd_oid8'::regclass \gset+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));+-- old value is an oid8 pointer, new value an oid pointer.+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;+ length | substr +--------+--------+ 100000 | aaa+(1 row)++-- replaced value must be gone, leaving a single value behind.+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;+ count +-------+ 1+(1 row)++-- reverse: old value is an oid pointer, new value an oid8 pointer.+TRUNCATE toastupd_oid;+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;+ length | substr +--------+--------+ 100000 | aaa+(1 row)++SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;+ count +-------+ 1+(1 row)++DROP TABLE toastupd_oid, toastupd_oid8;
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
-- corner case in packed-varlena handling: even though small, the compressed
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sqlindex ce5ea37a660c..229143f06ddd 100644--- a/src/test/regress/sql/compression.sql+++ b/src/test/regress/sql/compression.sql@@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2;
SELECT SUBSTR(f1, 200, 5) FROM cmdata2;
DROP TABLE cmdata2;
+-- pg_column_compression() with oid and oid8+CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid');+CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8');+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid;+SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8;+-- out-of-line and compressed.+TRUNCATE toastcomp_oid;+TRUNCATE toastcomp_oid8;+ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED;+ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;+ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128);+ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128);+INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000));+INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000));+SELECT pg_column_compression(f1) FROM toastcomp_oid;+SELECT pg_column_compression(f1) FROM toastcomp_oid8;+DROP TABLE toastcomp_oid, toastcomp_oid8;+
--test column type update varlena/non-varlena
CREATE TABLE cmdata2 (f1 int);
\d+ cmdata2
diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sqlindex 3e7459d29ff1..4d6a1e02359c 100644--- a/src/test/regress/sql/strings.sql+++ b/src/test/regress/sql/strings.sql@@ -727,6 +727,31 @@ ALTER TABLE toasttest_oid8 RESET (toast_value_type);
ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED;
ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED;
+-- UPDATE with out-of-line datum that belongs to another TOAST table.+CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid');+CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8');+ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL;+ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL;+SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class+ WHERE oid = 'toastupd_oid'::regclass \gset+SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class+ WHERE oid = 'toastupd_oid8'::regclass \gset+INSERT INTO toastupd_oid VALUES (repeat('a', 100000));+INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000));+-- old value is an oid8 pointer, new value an oid pointer.+UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid;+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8;+-- replaced value must be gone, leaving a single value behind.+SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0;+-- reverse: old value is an oid pointer, new value an oid8 pointer.+TRUNCATE toastupd_oid;+INSERT INTO toastupd_oid VALUES (repeat('c', 100000));+UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8;+SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid;+SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0;++DROP TABLE toastupd_oid, toastupd_oid8;+
-- test internally compressing datums
-- this tests compressing a datum to a very small size which exercises a
--
2.55.0
v15-0010-Add-support-for-TOAST-pointers-as-oid8.patch (text/plain, 51.0K) download
From a39c74131923f0e62e8339eff31f46adf439aa61 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 8 Sep 2026 11:48:29 +0900
Subject: [PATCH v15 10/11] Add support for TOAST pointers as oid8
This introduces a new varlena external pointer structure,
varatt_external_oid8, that carries a 64-bit value ID (Oid8) for
TOAST tables using oid8 as their chunk_id type.
XXX: Catalog version bump required.
---
src/include/access/detoast.h | 3 +
src/include/access/heaptoast.h | 16 +-
src/include/utils/rel.h | 2 +-
src/include/varatt.h | 93 ++++-
src/backend/access/common/detoast.c | 151 ++++++--
src/backend/access/common/toast_compression.c | 11 +-
src/backend/access/common/toast_internals.c | 353 ++++++++++++------
src/backend/access/heap/heaptoast.c | 17 +-
src/backend/access/table/toast_helper.c | 4 +
src/backend/catalog/toasting.c | 19 +-
.../replication/logical/reorderbuffer.c | 45 ++-
src/backend/utils/adt/varlena.c | 19 +-
doc/src/sgml/ref/create_table.sgml | 25 +-
doc/src/sgml/storage.sgml | 10 +-
contrib/amcheck/verify_heapam.c | 90 +++--
15 files changed, 631 insertions(+), 227 deletions(-)
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.hindex 2b1428b1e81c..f065ea034e98 100644--- a/src/include/access/detoast.h+++ b/src/include/access/detoast.h@@ -30,6 +30,9 @@ do { \
/* Size of an EXTERNAL datum that contains a standard TOAST pointer */
#define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid))
+/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */+#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8))+
/* Size of an EXTERNAL datum that contains an indirection pointer */
#define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect))
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.hindex aaae6e0fef69..b7ef69a8f56b 100644--- a/src/include/access/heaptoast.h+++ b/src/include/access/heaptoast.h@@ -69,13 +69,14 @@
/*
* When we store an oversize datum externally, we divide it into chunks
- * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must*- * be small enough that the completed toast-table tuple (including the- * ID and sequence fields and all overhead) will fit on a page.+ * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE+ * data bytes, depending on the chunk_id type of the TOAST table. These+ * numbers *must* be small enough that the completed toast-table tuple+ * (including the ID and sequence fields and all overhead) will fit on a page.
* The coding here sets the size on the theory that we want to fit
* EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page.
*
- * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb.+ * NB: Changing these values requires an initdb.
*/
#define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */
@@ -88,6 +89,13 @@
sizeof(int32) - \
VARHDRSZ)
+#define TOAST_OID8_MAX_CHUNK_SIZE \+ (EXTERN_TUPLE_MAX_SIZE - \+ MAXALIGN(SizeofHeapTupleHeader) - \+ sizeof(Oid8) - \+ sizeof(int32) - \+ VARHDRSZ)+
/* ----------
* heap_toast_insert_or_update -
*
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex b7da03446a94..8bec73d55794 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -386,7 +386,7 @@ typedef struct StdRdOptions
*/
#define RelationGetToastValueType(relation, defaulttarg) \
((relation)->rd_options ? \
- ((StdRdOptions *) (relation)->rd_options)->toast_value_type : defaulttarg)+ ((StdRdOptions *) (relation)->rd_options)->toast_value_type : (defaulttarg))
/*
* RelationGetFillFactor
diff --git a/src/include/varatt.h b/src/include/varatt.hindex 33979bbaaad1..1349a54371db 100644--- a/src/include/varatt.h+++ b/src/include/varatt.h@@ -38,6 +38,44 @@ typedef struct varatt_external_oid
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_oid;
+/*+ * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use+ * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid+ * except for the value ID, which is 8 bytes wide. The value ID is split+ * into two uint32 to force alignment.+ *+ * This struct must not contain any padding, because we sometimes compare+ * these pointers using memcmp.+ */+typedef struct varatt_external_oid8+{+ int32 va_rawsize; /* Original data size (includes header) */+ uint32 va_extinfo; /* External saved size (without header) and+ * compression method */+ uint32 va_valueid_lo; /* Low 32 bits of value ID */+ uint32 va_valueid_hi; /* High 32 bits of value ID */+ Oid va_toastrelid; /* RelID of TOAST table containing it */+} varatt_external_oid8;++StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) ==+ sizeof(varatt_external_oid8),+ "varatt_external_oid8 should have no padding");++/* Get/set the 64-bit value ID from the lo/hi halves */+static inline Oid8+VARATT_EXTERNAL_OID8_GET_VALUEID(varatt_external_oid8 toast_pointer)+{+ return ((Oid8) toast_pointer.va_valueid_lo) |+ (((Oid8) toast_pointer.va_valueid_hi) << 32);+}++static inline void+VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id)+{+ toast_pointer->va_valueid_lo = (uint32) id;+ toast_pointer->va_valueid_hi = (uint32) (id >> 32);+}+
/*
* These macros define the "saved size" portion of va_extinfo. Its remaining
* two high-order bits identify the compression method.
@@ -87,6 +125,7 @@ typedef enum vartag_external
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
+ VARTAG_ONDISK_OID8 = 4,
VARTAG_ONDISK_OID = 18
} vartag_external;
@@ -108,6 +147,8 @@ VARTAG_SIZE(vartag_external tag)
return sizeof(varatt_expanded);
else if (tag == VARTAG_ONDISK_OID)
return sizeof(varatt_external_oid);
+ else if (tag == VARTAG_ONDISK_OID8)+ return sizeof(varatt_external_oid8);
else
{
Assert(false);
@@ -360,10 +401,29 @@ VARATT_IS_EXTERNAL(const void *PTR)
/* Is varlena datum a pointer to on-disk toasted data? */
static inline bool
VARATT_IS_EXTERNAL_ONDISK(const void *PTR)
+{+ vartag_external tag;++ if (!VARATT_IS_EXTERNAL(PTR))+ return false;+ tag = VARTAG_EXTERNAL(PTR);+ return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8);+}++/* Is varlena datum a pointer to on-disk toasted data with 4-byte value ID? */+static inline bool+VARATT_IS_EXTERNAL_ONDISK_OID(const void *PTR)
{
return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID;
}
+/* Is varlena datum a pointer to on-disk toasted data with 8-byte value ID? */+static inline bool+VARATT_IS_EXTERNAL_ONDISK_OID8(const void *PTR)+{+ return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID8;+}+
/* Is varlena datum an indirect pointer? */
static inline bool
VARATT_IS_EXTERNAL_INDIRECT(const void *PTR)
@@ -519,16 +579,6 @@ VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer)
return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;
}
-/* Set size and compress method of an externally-stored varlena datum */-/* This has to remain a macro; beware multiple evaluations! */-#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \- do { \- Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \- (cm) == TOAST_LZ4_COMPRESSION_ID); \- ((toast_pointer).va_extinfo = \- (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \- } while (0)-
/*
* Testing whether an externally-stored value is compressed now requires
* comparing size stored in va_extinfo (the actual length of the external data)
@@ -543,4 +593,27 @@ VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer)
(Size) (toast_pointer.va_rawsize - VARHDRSZ);
}
+/*+ * Same for external Datums; but note argument is a struct+ * varatt_external_oid8.+ */+static inline Size+VARATT_EXTERNAL_OID8_GET_EXTSIZE(varatt_external_oid8 toast_pointer)+{+ return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK;+}++static inline uint32+VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(varatt_external_oid8 toast_pointer)+{+ return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS;+}++static inline bool+VARATT_EXTERNAL_OID8_IS_COMPRESSED(varatt_external_oid8 toast_pointer)+{+ return VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer) <+ (Size) (toast_pointer.va_rawsize - VARHDRSZ);+}+
#endif
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.cindex a50aea7b6b12..393135f1b9f3 100644--- a/src/backend/access/common/detoast.c+++ b/src/backend/access/common/detoast.c@@ -225,12 +225,32 @@ detoast_attr_slice(varlena *attr,
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
- varatt_external_oid toast_pointer;+ int32 extsize;+ uint32 compress_method;+ bool is_compressed;- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ /* Extract extsize and compression info from the appropriate pointer */+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer;++ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ extsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);+ compress_method = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);+ }+ else+ {+ varatt_external_oid toast_pointer;++ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ compress_method = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer);+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);+ }
/* fast path for non-compressed external datums */
- if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))+ if (!is_compressed)
return toast_fetch_datum_slice(attr, sliceoffset, slicelength);
/*
@@ -240,7 +260,7 @@ detoast_attr_slice(varlena *attr,
*/
if (slicelimit >= 0)
{
- int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ int32 max_size = extsize;
/*
* Determine maximum amount of compressed data needed for a prefix
@@ -251,14 +271,9 @@ detoast_attr_slice(varlena *attr,
* determine how much compressed data we need to be sure of being
* able to decompress the required slice.
*/
- if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) ==- TOAST_PGLZ_COMPRESSION_ID)+ if (compress_method == TOAST_PGLZ_COMPRESSION_ID)
max_size = pglz_maximum_compressed_size(slicelimit, max_size);
- /*- * Fetch enough compressed slices (compressed marker will get set- * automatically).- */
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
@@ -344,23 +359,47 @@ toast_fetch_datum(varlena *attr)
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;+ Oid8 valueid;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer;- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);+ toastrelid = toast_pointer.va_toastrelid;+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);- result = (varlena *) palloc(attrsize + VARHDRSZ);+ result = (varlena *) palloc(attrsize + VARHDRSZ);- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))- SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);+ else+ SET_VARSIZE(result, attrsize + VARHDRSZ);+ }
else
- SET_VARSIZE(result, attrsize + VARHDRSZ);+ {+ varatt_external_oid toast_pointer;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ toastrelid = toast_pointer.va_toastrelid;+ valueid = toast_pointer.va_valueid;++ result = (varlena *) palloc(attrsize + VARHDRSZ);++ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))+ SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ);+ else+ SET_VARSIZE(result, attrsize + VARHDRSZ);+ }
if (attrsize == 0)
return result; /* Probably shouldn't happen, but just in
@@ -369,10 +408,10 @@ toast_fetch_datum(varlena *attr)
/*
* Open the toast relation and its indexes
*/
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, 0, attrsize, result);
/* Close toast table */
@@ -398,23 +437,43 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
{
Relation toastrel;
varlena *result;
- varatt_external_oid toast_pointer;
int32 attrsize;
+ Oid toastrelid;+ Oid8 valueid;+ bool is_compressed;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums");
- /* Must copy to access aligned fields */- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ attrsize = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);+ toastrelid = toast_pointer.va_toastrelid;+ valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);+ is_compressed = VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer);+ }+ else+ {+ varatt_external_oid toast_pointer;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ toastrelid = toast_pointer.va_toastrelid;+ valueid = toast_pointer.va_valueid;+ is_compressed = VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer);+ }
/*
* It's nonsense to fetch slices of a compressed datum unless when it's a
* prefix -- this isn't lo_* we can't return a compressed datum which is
* meaningful to toast later.
*/
- Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset);-- attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ Assert(!is_compressed || 0 == sliceoffset);
if (sliceoffset >= attrsize)
{
@@ -427,7 +486,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
* space required by va_tcinfo, which is stored at the beginning as an
* int32 value.
*/
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0)+ if (is_compressed && slicelength > 0)
slicelength = slicelength + sizeof(int32);
/*
@@ -440,7 +499,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
result = (varlena *) palloc(slicelength + VARHDRSZ);
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))+ if (is_compressed)
SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ);
else
SET_VARSIZE(result, slicelength + VARHDRSZ);
@@ -449,10 +508,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
return result; /* Can save a lot of work at this point! */
/* Open the toast relation */
- toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);+ toastrel = table_open(toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,+ table_relation_fetch_toast_slice(toastrel, valueid,
attrsize, sliceoffset, slicelength,
result);
@@ -550,10 +609,20 @@ toast_raw_datum_size(Datum value)
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
/* va_rawsize is the size of the original datum -- including header */
- varatt_external_oid toast_pointer;+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer;- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);- result = toast_pointer.va_rawsize;+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ result = toast_pointer.va_rawsize;+ }+ else+ {+ varatt_external_oid toast_pointer;++ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ result = toast_pointer.va_rawsize;+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -610,10 +679,20 @@ toast_datum_size(Datum value)
* compressed or not. We do not count the size of the toast pointer
* ... should we?
*/
- varatt_external_oid toast_pointer;+ if (VARTAG_EXTERNAL(attr) == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer;- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);- result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ result = VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer);+ }+ else+ {+ varatt_external_oid toast_pointer;++ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer);+ }
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.cindex 3b1bd6519261..200cee6be705 100644--- a/src/backend/access/common/toast_compression.c+++ b/src/backend/access/common/toast_compression.c@@ -260,7 +260,16 @@ toast_get_compression_id(varlena *attr)
* the external toast pointer. If compressed inline, fetch it from the
* toast compression header.
*/
- if (VARATT_IS_EXTERNAL_ONDISK(attr))+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))+ {+ varatt_external_oid8 toast_pointer;++ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);++ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer))+ cmid = VARATT_EXTERNAL_OID8_GET_COMPRESS_METHOD(toast_pointer);+ }+ else if (VARATT_IS_EXTERNAL_ONDISK_OID(attr))
{
varatt_external_oid toast_pointer;
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex 2c180491c2ff..84c4361af23d 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -125,7 +125,6 @@ toast_save_datum(Relation rel, Datum value,
TupleDesc toasttupDesc;
CommandId mycid = GetCurrentCommandId(true);
varlena *result;
- varatt_external_oid toast_pointer;
int32 chunk_seq = 0;
char *data_p;
int32 data_todo;
@@ -133,9 +132,17 @@ toast_save_datum(Relation rel, Datum value,
int num_indexes;
int validIndex;
Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);
+ int32 max_chunk_size;++ /* Fields that will be assembled into the TOAST pointer at the end */+ int32 va_rawsize;+ uint32 va_extinfo;+ Oid8 va_valueid;+ Oid va_toastrelid;
Assert(!VARATT_IS_EXTERNAL(dval));
Assert(OidIsValid(toast_typid));
+ Assert(toast_typid == OIDOID || toast_typid == OID8OID);
/*
* Open the toast relation and its indexes. We can use the index to check
@@ -165,28 +172,32 @@ toast_save_datum(Relation rel, Datum value,
{
data_p = VARDATA_SHORT(dval);
data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT;
- toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */- toast_pointer.va_extinfo = data_todo;+ va_rawsize = data_todo + VARHDRSZ; /* as if not short */+ va_extinfo = data_todo;
}
else if (VARATT_IS_COMPRESSED(dval))
{
+ uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval);+
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;+ va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ;
/* set external size and compression method */
- VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo,- VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval));+ Assert(cmid == TOAST_PGLZ_COMPRESSION_ID ||+ cmid == TOAST_LZ4_COMPRESSION_ID);+ va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS);+
/* Assert that the numbers look like it's compressed */
- Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer));+ Assert((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ));
}
else
{
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
- toast_pointer.va_rawsize = VARSIZE(dval);- toast_pointer.va_extinfo = data_todo;+ va_rawsize = VARSIZE(dval);+ va_extinfo = data_todo;
}
/*
@@ -198,13 +209,14 @@ toast_save_datum(Relation rel, Datum value,
* if we have to substitute such an OID.
*/
if (OidIsValid(rel->rd_toastoid))
- toast_pointer.va_toastrelid = rel->rd_toastoid;+ va_toastrelid = rel->rd_toastoid;
else
- toast_pointer.va_toastrelid = RelationGetRelid(toastrel);+ va_toastrelid = RelationGetRelid(toastrel);
/*
- * Choose a new value to use as the value ID for this toast value, be it- * for OID or OID8 TOAST relations.+ * Choose a new value to use as the value ID for this toast value, and+ * determine the maximum chunk size based on the TOAST table's chunk_id+ * type.
*
* Normally we just choose an unused value within the toast table. But
* during table-rewriting operations where we are preserving an existing
@@ -213,84 +225,150 @@ toast_save_datum(Relation rel, Datum value,
* toast table, re-use its value ID. If we didn't have a prior external
* value (which is a corner case, but possible if the table's attstorage
* options have been changed), we have to pick a value ID that doesn't
- * conflict with either new or existing toast value IDs. If the TOAST- * table uses 8-byte value IDs, we should not really care much about- * that.+ * conflict with either new or existing toast value IDs. For Oid8 tables,+ * value conflicts are not a concern.
*/
- if (!OidIsValid(rel->rd_toastoid))+ if (toast_typid == OID8OID)
{
- /* normal case: just choose an unused OID */- if (toast_typid == OIDOID)- toast_pointer.va_valueid =- GetNewOidWithIndex(toastrel,- RelationGetRelid(toastidxs[validIndex]),- (AttrNumber) 1);- else if (toast_typid == OID8OID)- toast_pointer.va_valueid = GetNewObjectId8();+ if (!OidIsValid(rel->rd_toastoid))+ {+ /* normal case: just choose a new Oid8 */+ va_valueid = GetNewObjectId8();+ }
else
- Assert(false);+ {+ /* rewrite case: check to see if value was in old toast table */+ va_valueid = InvalidOid8;+ if (oldexternal != NULL)+ {+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));++ /*+ * Only consider the old pointer if its vartag matches the+ * kind of pointer we are about to build.+ */+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(oldexternal))+ {+ varatt_external_oid8 old_toast_pointer;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)+ {+ /*+ * This value came from the old toast table; reuse its+ * ID.+ */+ va_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(old_toast_pointer);++ /*+ * There is a corner case here: the table rewrite+ * might have to copy both live and recently-dead+ * versions of a row, and those versions could easily+ * reference the same toast value. When we copy the+ * second or later version of such a row, reusing the+ * OID will mean we select an OID that's already in the+ * new toast table. Check for that, and if so, just+ * fall through without writing the data again.+ *+ * While annoying and ugly-looking, this is a good+ * thing because it ensures that we wind up with only+ * one copy of the toast value when there is only one+ * copy in the old toast table. Before we detected+ * this case, we'd have made multiple copies, wasting+ * space; and what's worse, the copies belonging to+ * already-deleted heap tuples would not be reclaimed+ * by VACUUM.+ */+ if (toastrel_valueid_exists(toastrel, va_valueid))+ {+ /*+ * Match, so short-circuit the data storage loop+ * below.+ */+ data_todo = 0;+ }+ }+ }+ }+ if (va_valueid == InvalidOid8)+ {+ /*+ * New value. For Oid8 we just pick a new ID without worrying+ * about conflicts.+ */+ va_valueid = GetNewObjectId8();+ }+ }++ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;
}
else
{
- /* rewrite case: check to see if value was in old toast table */- toast_pointer.va_valueid = InvalidOid;- if (oldexternal != NULL)+ if (!OidIsValid(rel->rd_toastoid))
{
- varatt_external_oid old_toast_pointer;-- Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));- /* Must copy to access aligned fields */- VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);- if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)+ /* normal case: just choose an unused OID */+ va_valueid =+ GetNewOidWithIndex(toastrel,+ RelationGetRelid(toastidxs[validIndex]),+ (AttrNumber) 1);+ }+ else+ {+ /* rewrite case: check to see if value was in old toast table */+ va_valueid = InvalidOid;+ if (oldexternal != NULL)
{
- /* This value came from the old toast table; reuse its OID */- toast_pointer.va_valueid = old_toast_pointer.va_valueid;+ Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal));- /*- * There is a corner case here: the table rewrite might have- * to copy both live and recently-dead versions of a row, and- * those versions could easily reference the same toast value.- * When we copy the second or later version of such a row,- * reusing the OID will mean we select an OID that's already- * in the new toast table. Check for that, and if so, just- * fall through without writing the data again.- *- * While annoying and ugly-looking, this is a good thing- * because it ensures that we wind up with only one copy of- * the toast value when there is only one copy in the old- * toast table. Before we detected this case, we'd have made- * multiple copies, wasting space; and what's worse, the- * copies belonging to already-deleted heap tuples would not- * be reclaimed by VACUUM.- */- if (toastrel_valueid_exists(toastrel,- toast_pointer.va_valueid))+ /* Only reuse a pointer of the matching type; same as above */+ if (VARATT_IS_EXTERNAL_ONDISK_OID(oldexternal))
{
- /* Match, so short-circuit the data storage loop below */- data_todo = 0;+ varatt_external_oid old_toast_pointer;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal);+ if (old_toast_pointer.va_toastrelid == rel->rd_toastoid)+ {+ /*+ * This value came from the old toast table; reuse its+ * OID.+ */+ va_valueid = old_toast_pointer.va_valueid;++ /*+ * Corner case during table rewrite with multiple+ * versions of the same row. See above for details.+ */+ if (toastrel_valueid_exists(toastrel, va_valueid))+ {+ /*+ * Match, so short-circuit the data storage loop+ * below.+ */+ data_todo = 0;+ }+ }
}
}
- }- if (toast_pointer.va_valueid == InvalidOid)- {- /*- * new value; must choose a value that doesn't conflict in either- * old or new toast table.- */- if (toast_typid == OIDOID)+ if (va_valueid == InvalidOid)
{
+ /*+ * new value; must choose a value that doesn't conflict in+ * either old or new toast table.+ */
do
{
- toast_pointer.va_valueid =+ va_valueid =
GetNewOidWithIndex(toastrel,
RelationGetRelid(toastidxs[validIndex]),
(AttrNumber) 1);
} while (toastid_valueid_exists(rel->rd_toastoid,
- toast_pointer.va_valueid));+ va_valueid));
}
- else if (toast_typid == OID8OID)- toast_pointer.va_valueid = GetNewObjectId8();
}
++ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
}
/*
@@ -314,15 +392,15 @@ toast_save_datum(Relation rel, Datum value,
/*
* Calculate the size of this chunk
*/
- chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo);+ chunk_size = Min(max_chunk_size, data_todo);
/*
* Build a tuple and store it
*/
- if (toast_typid == OIDOID)- t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid);- else if (toast_typid == OID8OID)- t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid);+ if (toast_typid == OID8OID)+ t_values[0] = ObjectId8GetDatum(va_valueid);+ else+ t_values[0] = ObjectIdGetDatum((Oid) va_valueid);
t_values[1] = Int32GetDatum(chunk_seq++);
SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ);
memcpy(VARDATA(&chunk_data), data_p, chunk_size);
@@ -376,11 +454,34 @@ toast_save_datum(Relation rel, Datum value,
table_close(toastrel, NoLock);
/*
- * Create the TOAST pointer value that we'll return+ * Create the TOAST pointer value that we'll return.
*/
- result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);- SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);- memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));+ if (toast_typid == OID8OID)+ {+ varatt_external_oid8 toast_pointer;++ toast_pointer.va_rawsize = va_rawsize;+ toast_pointer.va_extinfo = va_extinfo;+ VARATT_EXTERNAL_OID8_SET_VALUEID(&toast_pointer, va_valueid);+ toast_pointer.va_toastrelid = va_toastrelid;++ result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE);+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8);+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));+ }+ else+ {+ varatt_external_oid toast_pointer;++ toast_pointer.va_rawsize = va_rawsize;+ toast_pointer.va_extinfo = va_extinfo;+ toast_pointer.va_valueid = (Oid) va_valueid;+ toast_pointer.va_toastrelid = va_toastrelid;++ result = (varlena *) palloc(TOAST_OID_POINTER_SIZE);+ SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID);+ memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer));+ }
return PointerGetDatum(result);
}
@@ -395,7 +496,6 @@ void
toast_delete_datum(Relation rel, Datum value, bool is_speculative)
{
varlena *attr = (varlena *) DatumGetPointer(value);
- varatt_external_oid toast_pointer;
Relation toastrel;
Relation *toastidxs;
ScanKeyData toastkey;
@@ -403,42 +503,75 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative)
HeapTuple toasttup;
int num_indexes;
int validIndex;
- Oid toast_typid;+ Oid toastrelid;+ vartag_external tag;
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
return;
- /* Must copy to access aligned fields */- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);-
/*
- * Open the toast relation and its indexes+ * Determine the pointer type from the datum's vartag and extract the+ * toast relation OID and value ID accordingly. The vartag tells us+ * everything we need - no TOAST table schema lookup required.
*/
- toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock);- toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;- Assert(toast_typid == OIDOID || toast_typid == OID8OID);+ tag = VARTAG_EXTERNAL(attr);- /* Fetch valid relation used for process */- validIndex = toast_open_indexes(toastrel,- RowExclusiveLock,- &toastidxs,- &num_indexes);+ if (tag == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer8;- /*- * Setup a scan key to find chunks with matching va_valueid- */- if (toast_typid == OIDOID)+ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);+ toastrelid = toast_pointer8.va_toastrelid;++ /*+ * Open the toast relation and its indexes+ */+ toastrel = table_open(toastrelid, RowExclusiveLock);++ /* Fetch valid relation used for process */+ validIndex = toast_open_indexes(toastrel,+ RowExclusiveLock,+ &toastidxs,+ &num_indexes);++ /*+ * Setup a scan key to find chunks with matching va_valueid+ */+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OID8EQ,+ ObjectId8GetDatum(VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8)));+ }+ else+ {+ varatt_external_oid toast_pointer;++ Assert(tag == VARTAG_ONDISK_OID);++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toastrelid = toast_pointer.va_toastrelid;++ /*+ * Open the toast relation and its indexes+ */+ toastrel = table_open(toastrelid, RowExclusiveLock);++ /* Fetch valid relation used for process */+ validIndex = toast_open_indexes(toastrel,+ RowExclusiveLock,+ &toastidxs,+ &num_indexes);++ /*+ * Setup a scan key to find chunks with matching va_valueid+ */
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(toast_pointer.va_valueid));
- else if (toast_typid == OID8OID)- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OID8EQ,- ObjectId8GetDatum(toast_pointer.va_valueid));- else- Assert(false);+ }
/*
* Find all the chunks. (We don't actually care whether we see them in
@@ -499,18 +632,16 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid)
/*
* Setup a scan key to find chunks with matching va_valueid
*/
- if (toast_typid == OIDOID)- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(valueid));- else if (toast_typid == OID8OID)+ if (toast_typid == OID8OID)
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
ObjectId8GetDatum(valueid));
else
- Assert(false);+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum((Oid) valueid));
/*
* Is there any such chunk?
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.cindex a93e98788da7..31e6115de201 100644--- a/src/backend/access/heap/heaptoast.c+++ b/src/backend/access/heap/heaptoast.c@@ -650,7 +650,10 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid;
Assert(toast_typid == OIDOID || toast_typid == OID8OID);
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;+ if (toast_typid == OID8OID)+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;+ else+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;
totalchunks = ((attrsize - 1) / max_chunk_size) + 1;
startchunk = sliceoffset / max_chunk_size;
@@ -658,18 +661,16 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize,
Assert(endchunk <= totalchunks);
/* Set up a scan key to fetch from the index. */
- if (toast_typid == OIDOID)- ScanKeyInit(&toastkey[0],- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(valueid));- else if (toast_typid == OID8OID)+ if (toast_typid == OID8OID)
ScanKeyInit(&toastkey[0],
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
ObjectId8GetDatum(valueid));
else
- Assert(false);+ ScanKeyInit(&toastkey[0],+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum((Oid) valueid));
/*
* No additional condition if fetching all chunks. Otherwise, use an
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.cindex 6f2d691cc681..6c7257f5f94b 100644--- a/src/backend/access/table/toast_helper.c+++ b/src/backend/access/table/toast_helper.c@@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc)
/*
* If the old value is stored on disk, check if it has changed so
* we have to delete it later.
+ *+ * Note that TOAST pointers could have different vartags, for oid+ * or oid8, and these can have a different sizes.
*/
if (att->attlen == -1 && !ttc->ttc_oldisnull[i] &&
VARATT_IS_EXTERNAL_ONDISK(old_value))
{
if (ttc->ttc_isnull[i] ||
!VARATT_IS_EXTERNAL_ONDISK(new_value) ||
+ VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) ||
memcmp(old_value, new_value,
VARSIZE_EXTERNAL(old_value)) != 0)
{
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.cindex 66d483bb1be3..57ebaec69c78 100644--- a/src/backend/catalog/toasting.c+++ b/src/backend/catalog/toasting.c@@ -167,10 +167,21 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
return false;
value_type = RelationGetToastValueType(rel, STDRD_OPTION_TOAST_VALUE_TYPE_OID);
- if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID)- toast_chunkid_typid = OIDOID;- else if (value_type == STDRD_OPTION_TOAST_VALUE_TYPE_OID8)- toast_chunkid_typid = OID8OID;++ /* no default clause to catch new values added */+ switch (value_type)+ {+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID:+ toast_chunkid_typid = OIDOID;+ break;+ case STDRD_OPTION_TOAST_VALUE_TYPE_OID8:+ toast_chunkid_typid = OID8OID;+ break;+ case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID:+ elog(ERROR, "unexpected toast_value_type value %d",+ value_type);+ break;+ }
}
else
{
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.cindex dd4b6e8f7f45..086ef31e4f2f 100644--- a/src/backend/replication/logical/reorderbuffer.c+++ b/src/backend/replication/logical/reorderbuffer.c@@ -5176,12 +5176,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* va_rawsize is the size of the original datum -- including header */
varatt_external_oid toast_pointer;
+ varatt_external_oid8 toast_pointer8;
varatt_indirect redirect_pointer;
varlena *new_datum = NULL;
varlena *reconstructed;
dlist_iter it;
Size data_done = 0;
Oid8 toast_valueid;
+ int32 rawsize;
if (attr->attisdropped)
continue;
@@ -5198,11 +5200,21 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
varlena_pointer = (varlena *) DatumGetPointer(attrs[natt]);
/* no need to do anything if the tuple isn't external */
- if (!VARATT_IS_EXTERNAL(varlena_pointer))+ if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer))
continue;
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);- toast_valueid = toast_pointer.va_valueid;+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))+ {+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, varlena_pointer);+ toast_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);+ rawsize = toast_pointer8.va_rawsize;+ }+ else+ {+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer);+ toast_valueid = toast_pointer.va_valueid;+ rawsize = toast_pointer.va_rawsize;+ }
/*
* Check whether the toast tuple changed, replace if so.
@@ -5220,7 +5232,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
free[natt] = true;
- reconstructed = palloc0(toast_pointer.va_rawsize);+ reconstructed = palloc0(rawsize);
ent->reconstructed = reconstructed;
@@ -5245,13 +5257,28 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
VARSIZE(chunk) - VARHDRSZ);
data_done += VARSIZE(chunk) - VARHDRSZ;
}
- Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));- /* make sure its marked as compressed or not */- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))- SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);+ /* Verify size and set compression status based on pointer type */+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(varlena_pointer))+ {+ Assert(data_done == VARATT_EXTERNAL_OID8_GET_EXTSIZE(toast_pointer8));++ /* make sure its marked as compressed or not */+ if (VARATT_EXTERNAL_OID8_IS_COMPRESSED(toast_pointer8))+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);+ else+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);+ }
else
- SET_VARSIZE(reconstructed, data_done + VARHDRSZ);+ {+ Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer));++ /* make sure its marked as compressed or not */+ if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))+ SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ);+ else+ SET_VARSIZE(reconstructed, data_done + VARHDRSZ);+ }
memset(&redirect_pointer, 0, sizeof(redirect_pointer));
redirect_pointer.pointer = reconstructed;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.cindex 1e264904a6e1..e104a6308d65 100644--- a/src/backend/utils/adt/varlena.c+++ b/src/backend/utils/adt/varlena.c@@ -4261,7 +4261,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
{
int typlen;
varlena *attr;
- varatt_external_oid toast_pointer;+ Oid8 result;
/* On first call, get the input type's typlen, and save at *fn_extra */
if (fcinfo->flinfo->fn_extra == NULL)
@@ -4288,9 +4288,22 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS)
if (!VARATT_IS_EXTERNAL_ONDISK(attr))
PG_RETURN_NULL();
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ if (VARATT_IS_EXTERNAL_ONDISK_OID8(attr))+ {+ varatt_external_oid8 toast_pointer;- PG_RETURN_OID8(toast_pointer.va_valueid);+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ result = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer);+ }+ else+ {+ varatt_external_oid toast_pointer;++ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ result = toast_pointer.va_valueid;+ }++ PG_RETURN_OID8(result);
}
/*
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgmlindex 3ed8f3581177..158f79e7135f 100644--- a/doc/src/sgml/ref/create_table.sgml+++ b/doc/src/sgml/ref/create_table.sgml@@ -1661,14 +1661,23 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- The toast_value_type specifies the attribute type of- <literal>chunk_id</literal> used when initially creating a toast- relation for this table.- By default this parameter is <literal>oid</literal>, to assign- <type>oid</type> as attribute type to <literal>chunk_id</literal>.- This parameter can be set to <type>oid8</type> to use <type>oid8</type>- as attribute type for <literal>chunk_id</literal>.- This parameter cannot be set for TOAST tables.+ Specifies the attribute type of <literal>chunk_id</literal> to use when+ creating a <acronym>TOAST</acronym> relation for this table. The+ default is <literal>oid</literal>; <literal>oid8</literal> assigns+ <type>oid8</type> instead, which allows a larger number of distinct+ <acronym>TOAST</acronym>ed values at the price of four extra bytes per+ out-of-line pointer. This parameter cannot be set for+ <acronym>TOAST</acronym> tables.+ </para>+ <para>+ The value is only consulted when the <acronym>TOAST</acronym> relation+ is created. Changing it afterwards has no effect on an existing+ <acronym>TOAST</acronym> relation, and a table rewrite such as+ <link linkend="sql-cluster"><command>CLUSTER</command></link> or+ <link linkend="sql-vacuum"><command>VACUUM FULL</command></link>+ preserves the type already in use. A dump and restore, on the other+ hand, recreates the <acronym>TOAST</acronym> relation and therefore+ applies the current value of this parameter.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 21bd67c11ea0..3561a6a67ece 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -417,7 +417,9 @@ described in more detail below.
<para>
Out-of-line values are divided (after compression if used) into chunks of at
-most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen+most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or+<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the+<structfield>chunk_id</structfield> type (by default this value is chosen
so that four chunk rows will fit on a page, making it about 2000 bytes).
Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table
belonging to the owning table. Every
@@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk
logical datum size (original uncompressed data length), physical stored size
(different if compression was applied), and the compression method used, if
any. Allowing for the varlena header bytes,
-the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18-bytes regardless of the actual size of the represented value.+the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18+bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes+when using an OID8 as <structfield>chunk_id</structfield>, regardless of the+actual size of the represented value.
</para>
<para>
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex 42f409548e1d..86dc9cf51a93 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -75,7 +75,9 @@ typedef enum SkipPages
*/
typedef struct ToastedAttribute
{
- varatt_external_oid toast_pointer;+ vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */+ Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */+ uint32 va_extinfo; /* external size and compression method */
BlockNumber blkno; /* block in main table */
OffsetNumber offnum; /* offset in main table */
AttrNumber attnum; /* attribute in main table */
@@ -1566,9 +1568,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx,
int32 max_chunk_size;
Oid8 toast_valueid;
- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;+ toast_valueid = ta->va_valueid;++ max_chunk_size = ta->tag == VARTAG_ONDISK_OID8+ ? TOAST_OID8_MAX_CHUNK_SIZE+ : TOAST_OID_MAX_CHUNK_SIZE;
last_chunk_seq = (extsize - 1) / max_chunk_size;
- toast_valueid = ta->toast_pointer.va_valueid;
/* Sanity-check the sequence number. */
chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2,
@@ -1672,8 +1677,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
char *tp; /* pointer to the tuple data */
uint16 infomask;
Oid8 toast_pointer_valueid;
+ int32 va_rawsize;+ uint32 va_extinfo;
CompactAttribute *thisatt;
- varatt_external_oid toast_pointer;+ vartag_external va_tag_value;
infomask = ctx->tuphdr->t_infomask;
thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum);
@@ -1732,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx)
{
uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);
- if (va_tag != VARTAG_ONDISK_OID)+ if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8)
{
report_corruption(ctx,
psprintf("toasted attribute has unexpected TOAST tag %u",
@@ -1777,26 +1784,46 @@ check_tuple_attribute(HeapCheckContext *ctx)
/* It is external, and we're looking at a page on disk */
/*
- * Must copy attr into toast_pointer for alignment considerations+ * Must copy attr into toast_pointer for alignment considerations.+ * Branch on the tag to determine which pointer type to extract.
*/
- VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);- toast_pointer_valueid = toast_pointer.va_valueid;+ va_tag_value = VARTAG_EXTERNAL(attr);+ if (va_tag_value == VARTAG_ONDISK_OID8)+ {+ varatt_external_oid8 toast_pointer8;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer8, attr);+ toast_pointer_valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer8);+ va_rawsize = toast_pointer8.va_rawsize;+ va_extinfo = toast_pointer8.va_extinfo;+ }+ else+ {+ varatt_external_oid toast_pointer;++ /* Must copy to access aligned fields */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);+ toast_pointer_valueid = toast_pointer.va_valueid;+ va_rawsize = toast_pointer.va_rawsize;+ va_extinfo = toast_pointer.va_extinfo;+ }
/* Toasted attributes too large to be untoasted should never be stored */
- if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)+ if (va_rawsize > VARLENA_SIZE_LIMIT)
report_corruption(ctx,
psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d",
toast_pointer_valueid,
- toast_pointer.va_rawsize,+ va_rawsize,
VARLENA_SIZE_LIMIT));
- if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer))+ if ((va_extinfo & VARLENA_EXTSIZE_MASK) < (Size) (va_rawsize - VARHDRSZ))
{
ToastCompressionId cmid;
bool valid = false;
/* Compressed attributes should have a valid compression method */
- cmid = TOAST_COMPRESS_METHOD(&toast_pointer);+ cmid = va_extinfo >> VARLENA_EXTSIZE_BITS;
switch (cmid)
{
/* List of all valid compression method IDs */
@@ -1850,7 +1877,10 @@ check_tuple_attribute(HeapCheckContext *ctx)
ta = palloc0_object(ToastedAttribute);
- VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr);+ /* The pointer has already been decoded above, just reuse it */+ ta->tag = va_tag_value;+ ta->va_valueid = toast_pointer_valueid;+ ta->va_extinfo = va_extinfo;
ta->blkno = ctx->blkno;
ta->offnum = ctx->offnum;
ta->attnum = ctx->attnum;
@@ -1878,29 +1908,33 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
int32 last_chunk_seq;
int32 max_chunk_size;
Oid8 toast_valueid;
- Oid toast_typid;- toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid;- max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;-- extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer);- last_chunk_seq = (extsize - 1) / max_chunk_size;+ toast_valueid = ta->va_valueid;+ extsize = ta->va_extinfo & VARLENA_EXTSIZE_MASK;
/*
* Setup a scan key to find chunks in toast table with matching va_valueid
*/
- if (toast_typid == OIDOID)- ScanKeyInit(&toastkey,- (AttrNumber) 1,- BTEqualStrategyNumber, F_OIDEQ,- ObjectIdGetDatum(ta->toast_pointer.va_valueid));- else if (toast_typid == OID8OID)+ if (ta->tag == VARTAG_ONDISK_OID8)+ {+ max_chunk_size = TOAST_OID8_MAX_CHUNK_SIZE;+
ScanKeyInit(&toastkey,
(AttrNumber) 1,
BTEqualStrategyNumber, F_OID8EQ,
- ObjectId8GetDatum(ta->toast_pointer.va_valueid));+ ObjectId8GetDatum(toast_valueid));+ }
else
- Assert(false);+ {+ max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE;++ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum((Oid) toast_valueid));+ }++ last_chunk_seq = (extsize - 1) / max_chunk_size;
/*
* Check if any chunks for this toasted object exist in the toast table,
@@ -1920,8 +1954,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta)
}
systable_endscan_ordered(toastscan);
- toast_valueid = ta->toast_pointer.va_valueid;-
if (!found_toasttup)
report_toast_corruption(ctx, ta,
psprintf("toast value " OID8_FORMAT " not found in toast table",
--
2.55.0
v15-0011-Fix-toast_tuple_find_biggest_attribute-for-OID8-.patch (text/plain, 6.7K) download
From 9a2cdce244f142734dbc925abbe769abdecd8a28 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 7 Sep 2026 15:32:34 +0900
Subject: [PATCH v15 11/11] Fix toast_tuple_find_biggest_attribute() for OID8
TOAST values
This routine is used to decide if largest varlena attributes should be
compressed or not, but stuck its decisions to a hardcoded
TOAST_OID_POINTER_SIZE, which would be amiss when dealing with an oid8
TOAST value.
This information is added to the parent Relation, and can be asked on a
per-call basis of a new RelationGetToastChunkIdType().
---
src/include/utils/rel.h | 9 +++++
src/include/utils/relcache.h | 1 +
src/backend/access/common/toast_internals.c | 3 +-
src/backend/access/table/toast_helper.c | 18 ++++++++--
src/backend/utils/cache/relcache.c | 37 +++++++++++++++++++++
5 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex 8bec73d55794..99d32cbcb21f 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -250,6 +250,15 @@ typedef struct RelationData
*/
Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */
+ /*+ * Type OID of the "chunk_id" column of this relation's TOAST table, i.e.+ * OIDOID or OID8OID.+ *+ * This data is filled on demand by RelationGetToastChunkIdType(), not+ * at relcache build time, so as to save in syscache lookups.+ */+ Oid rd_toastchunkidtype;+
bool pgstat_enabled; /* should relation stats be counted */
/* use "struct" here to avoid needing to include pgstat.h: */
struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.hindex 89c27aa1529f..e17c84908332 100644--- a/src/include/utils/relcache.h+++ b/src/include/utils/relcache.h@@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok);
extern Oid RelationGetReplicaIndex(Relation relation);
+extern Oid RelationGetToastChunkIdType(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
extern List *RelationGetDummyIndexExpressions(Relation relation);
extern List *RelationGetIndexPredicate(Relation relation);
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.cindex 84c4361af23d..5a7f63253f84 100644--- a/src/backend/access/common/toast_internals.c+++ b/src/backend/access/common/toast_internals.c@@ -25,7 +25,6 @@
#include "utils/fmgroids.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-#include "utils/lsyscache.h"
static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid);
static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid);
@@ -131,7 +130,7 @@ toast_save_datum(Relation rel, Datum value,
Pointer dval = DatumGetPointer(value);
int num_indexes;
int validIndex;
- Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1);+ Oid toast_typid = RelationGetToastChunkIdType(rel);
int32 max_chunk_size;
/* Fields that will be assembled into the TOAST pointer at the end */
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.cindex 6c7257f5f94b..3235d033ea9b 100644--- a/src/backend/access/table/toast_helper.c+++ b/src/backend/access/table/toast_helper.c@@ -175,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc)
* The column must have attstorage EXTERNAL or EXTENDED if check_main is
* false, and must have attstorage MAIN if check_main is true.
*
- * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE);- * if not, no benefit is to be expected by compressing it.+ * The column must be larger than the TOAST pointer that would replace it;+ * if not, no benefit is to be expected by compressing it. Note that this+ * choice depends on the TOAST value type, oid or oid8.
*
* The return value is the index of the biggest suitable column, or
* -1 if there is none.
@@ -188,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
int biggest_attno = -1;
- int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);+ int32 biggest_size;
int32 skip_colflags = TOASTCOL_IGNORE;
int i;
+ /*+ * Size of the TOAST pointer this relation would use. A relation without+ * a TOAST table cannot have any of its attributes moved out-of-line, but+ * it can still have some of them compressed. Fall back to the oid size+ * in that case.+ */+ if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID)+ biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE);+ else+ biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE);+
if (for_compression)
skip_colflags |= TOASTCOL_INCOMPRESSIBLE;
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.cindex f475d7039772..65dfdbda100a 100644--- a/src/backend/utils/cache/relcache.c+++ b/src/backend/utils/cache/relcache.c@@ -1205,6 +1205,9 @@ retry:
relation->rd_fkeylist = NIL;
relation->rd_fkeyvalid = false;
+ /* TOAST type data is not loaded till asked for */+ relation->rd_toastchunkidtype = InvalidOid;+
/* partitioning data is not loaded till asked for */
relation->rd_partkey = NULL;
relation->rd_partkeycxt = NULL;
@@ -5093,6 +5096,39 @@ RelationGetReplicaIndex(Relation relation)
return relation->rd_replidindex;
}
+/*+ * RelationGetToastChunkIdType -- get the type of the relation's TOAST+ * table "chunk_id" column+ *+ * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST+ * table.+ */+Oid+RelationGetToastChunkIdType(Relation relation)+{+ Oid toastrelid = relation->rd_rel->reltoastrelid;+ Oid typid;++ /* Quick exit if we already computed the value */+ if (OidIsValid(relation->rd_toastchunkidtype))+ return relation->rd_toastchunkidtype;++ /* Nothing to report without a TOAST table */+ if (!OidIsValid(toastrelid))+ return InvalidOid;++ typid = get_atttype(toastrelid, 1);+ if (!OidIsValid(typid))+ elog(ERROR, "cache lookup failed for TOAST relation %u",+ toastrelid);+ if (typid != OIDOID && typid != OID8OID)+ elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u",+ typid, toastrelid);++ relation->rd_toastchunkidtype = typid;+ return typid;+}+
/*
* RelationGetIndexExpressions -- get the index expressions for an index
*
@@ -6530,6 +6566,7 @@ load_relcache_init_file(bool shared)
rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId;
rel->rd_droppedSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_toastchunkidtype = InvalidOid;
rel->pgstat_info = NULL;
/*
--
2.55.0
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: michael@paquier.xyz, bharath.rupireddyforpostgres@gmail.com, nagata@sraoss.co.jp, pgsql-hackers@lists.postgresql.org
Subject: Re: Support for 8-byte TOAST values, round two
In-Reply-To: <ap-F89TpWEq324ij@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox